From the factory, the FPGA cards have only a “gold” bitfile in flash with the “user” partition of flash being blank. The “gold” bitfile has a narrow PCIe memory window for BAR1 and BAR2 which is insufficient for the ESnet Smartnic platform. Fixing this requires a one-time flash programming step to install an ESnet Smartnic bitfile into the FPGA “user” partition in flash. This initial setup is done using the JTAG.
Ensure that any running sn-stack instances have been stopped so that they don’t interfere with the flash programming operation.
docker compose down -v --remove-orphans
Start the flash rescue service to program an ESnet Smartnic bitfile into the FPGA card “user” partition using the JTAG interface. This takes approximately 20 minutes. This process should not be interrupted.
docker compose --profile smartnic-flash run --rm smartnic-flash-rescue
This will:
Clean up by bringing down the running stack after flash writing has completed.
docker compose down -v --remove-orphans
Perform a cold-boot (power cycle) of the server hosting the FPGA card
It is essential that this is a proper power cycle and not simply a warm reboot. Specifically do not use shutdown -r now but rather use something like ipmitool chassis power cycle. Failure to perform a cold-boot here will result in an unusable card.
Start up the full firmware docker stack like this
docker compose up -d
docker compose logs smartnic-hw
The firmware runtime environment exists inside of the smartnic-fw container. Here, we exec a shell inside of that container and have a look around.
docker compose exec smartnic-fw bash
sn-cli dev version
regio syscfg
If this prints wrong values or all ones, make sure you don’t have the COMPOSE_PROFILES set to smartnic-mgr-dpdk-manual in your sn-stack/.env file. Otherwise, you will not be able to access any sn-cli tools unless pktgen is running.
The instructions in this section are used to update the Smartnic flash image from an already working Smartnic environment. This update step is optional and only required if you want to change the contents of the FPGA card flash. Normally, the “RAM” of the FPGA is loaded using JTAG during stack startup.
NOTE This will not work for the very first time ever programming the flash. See “Converting from factory flash image to ESnet Smartnic flash image” section above for first-time setup.
Start up a any properly configured stack which will allow us to write the flash using a fast algorithm over PCIe.
docker compose up -d
Confirm that PCIe register IO is working in your stack by querying the version registers.
docker compose exec smartnic-fw sn-cli dev version
Confirm that the “DNA” register is not showing 0xfffff… as its contents.
Start the flash update service to write the currently active FPGA bitfile into the persistent flash on the FPGA card. This takes approximately 7-8 minutes. This process should not be interrupted.
docker compose --profile smartnic-flash run --rm smartnic-flash-update
Bring down the running stack after flash writing has completed.
docker compose down -v --remove-orphans
The instructions in this section are used to remove the Smartnic flash image from an already working Smartnic environment. This removal step is optional and only required if you want to reset the contents of the FPGA card flash back to the factory bitfile. If you want to keep using the card as an ESnet Smartnic, do not perform these operations or you’ll have to re-do the “Converting from factory flash image to ESnet Smartnic flash image” section above.
Start up a any properly configured stack which will allow us to write the flash using a fast algorithm over PCIe.
docker compose up -d
Confirm that PCIe register IO is working in your stack by querying the version registers.
docker compose exec smartnic-fw sn-cli dev version
Confirm that the “DNA” register is not showing 0xfffff… as its contents.
Start the flash remove service to erase the ESnet Smartnic image from the “user” partition of the FPGA card flash. This takes less than 1 minute. This process should not be interrupted.
docker compose --profile smartnic-flash run --rm smartnic-flash-remove
Bring down the running stack after flash reset is completed.
docker compose down -v --remove-orphans
Note: If you want to flash the golden (recovery) image and it is not working, you can use vivado_lab with the following commands:
vivado_lab \
-nolog \
-nojournal \
-tempDir /tmp/ \
-mode batch \
-notrace \
-quiet \
-source /scripts/program_flash.tcl \
-tclargs "$HW_SERVER_URL" "$HW_TARGET_SERIAL" "/scripts/revert_to_golden.mcs"
If you don’t know how to set it up, you can go to sn-stack/smartnic-hw/scripts and put the “revert to golden” image there. For more information on how to get the golden image, you can refer to this link.
After placing the image there, you can modify the program_flash.sh script and change:
vivado_lab \
-nolog \
-nojournal \
-tempDir /tmp/ \
-mode batch \
-notrace \
-quiet \
-source /scripts/program_flash.tcl \
-tclargs "$HW_SERVER_URL" "$HW_TARGET_SERIAL" "$MCSFILE_PATH
To:
vivado_lab \
-nolog \
-nojournal \
-tempDir /tmp/ \
-mode batch \
-notrace \
-quiet \
-source /scripts/program_flash.tcl \
-tclargs "$HW_SERVER_URL" "$HW_TARGET_SERIAL" "/scripts/revert_to_golden.mcs"
After a cold reboot, you will see the cards are back to the golden image.
Important notice: The golden image will not make the cards appear in XRT, as XRT needs an “XRT-friendly” shell. However, it’ll make xbmgmt see the card, and from there, you can flash a new platform that works with XRT. For more info on the golden image, you can refer to this documentation.
The sn-cli tool provides subcommands to help you accomplish many common tasks for inspecting and configuring the smartnic platform components.
All commands described below are expected to be executed within the smartnic-fw container environment. Use this command to enter the appropriate environment.
docker compose exec smartnic-fw bash
The sn-cli tool will automatically look for an environment variable called SN_CLI_SLOTADDR which can be set to the PCIe BDF address of the device that you would like to interract with. In the smartnic-fw container, this value will already be set for you.
This will show information about the device such as the build version, build date/time and temperature.
root@smartnic-fw:/# sn-cli dev version
Device Version Info
DNA: 0x40020000012306a21c10c285
USR_ACCESS: 0x000086d3 (34515)
BUILD_STATUS: 0x04130920
root@smartnic-fw:/# sn-cli dev temp
Temperature Monitors
FPGA SLR0: 45.551 (deg C)
The USR_ACCESS value is typically the unique build pipeline number that produced the embedded FPGA bitfile.
The BUILD_STATUS value holds an encoded date/time (Aug 30 at 05:32am) which is when the embedded FPGA bitfile build was started.
The DNA value holds the factory programmed unique ID of the FPGA
Enable/Disable one or more (or all by default) 100G MAC interfaces using these commands:
sn-cli cmac enable
sn-cli cmac disable
sn-cli cmac -p 0 enable
sn-cli cmac -p 1 disable
Enabling a CMAC interface allows frames to pass (Rx/Tx) at the MAC layer. These commands do not affect whether the underlying physical layer (PHY) is operational.
Display the current MAC and PHY status of one or more (or all by default) 100G MAC interfaces using these commands:
root@smartnic-fw:/# sn-cli cmac status
CMAC0
Tx (MAC ENABLED/PHY UP)
Rx (MAC ENABLED/PHY UP)
CMAC1
Tx (MAC ENABLED/PHY UP)
Rx (MAC ENABLED/PHY DOWN)
In the example output above, CMAC0 PHY layer is UP in both the Tx and Rx directions. The MAC is fully enabled. This link is operational and should be passing packets normally.
In the example output above, CMAC1 PHY layer is DOWN in the Rx (receive) direction. Possible causes for this are:
A more detailed status can also be displayed using the --verbose option. Note that the --verbose option is a global option and thus must be positioned before the cmac subcommand.
root@smartnic-fw:/# sn-cli --verbose cmac -p 1 status
CMAC1
Tx (MAC ENABLED/PHY UP)
tx_local_fault 0
Rx (MAC ENABLED/PHY DOWN)
rx_got_signal_os 0
rx_bad_sfd 0
rx_bad_preamble 0
rx_test_pattern_mismatch 0
rx_received_local_fault 0
rx_internal_local_fault 1
rx_local_fault 1
rx_remote_fault 0
rx_hi_ber 0
rx_aligned_err 0
rx_misaligned 0
rx_aligned 0
rx_status 0
Display summary statistics for packets Rx’d and Tx’d from CMAC ports
root@smartnic-fw:/# sn-cli cmac stats
CMAC0: TX 0 RX 0 RX-DISC 0 RX-ERR 0
CMAC1: TX 0 RX 0 RX-DISC 0 RX-ERR 0
Note: The CMAC counters are only cleared/reset when the FPGA is reprogrammed.
The QDMA block is responsible for managing all DMA queues used for transferring packets and/or events bidirectionally between the U280 card and the Host CPU over the PCIe bus. In order for any DMA transfers to be allowed on either of the PCIe Physical Functions (PF), an appropriate number of DMA Queue IDs must be provisioned. This can be done using the qdma subcommand.
Configure the number of queues allocated to each of the PCIe Physical Functions
sn-cli qdma setqs 1 1
This assigns 1 QID to PF0 and 1 QIDs to PF1. The setqs subcommand also takes care of configuring the RSS entropy -> QID map with an equal weighted distribution of all allocated queues. If you’re unsure of how many QIDs to allocate, using 1 1 here is your best choice.
Inspect the configuration of the QDMA block
sn-cli qdma status
Packet, byte and error counters are tracked for packets heading between the QDMA engine and the user application. You can display them with this command:
sn-cli qdma stats
Refer to the open-nic-shell documentation for an explanation of exactly where in the FPGA design these statistics are measured.
The smartnic platform implements monitoring points in the datapath at various locations. You an inspect these counters using this command:
sn-cli probe stats
Refer to the esnet-smartnic-hw documentation for an explanation of exactly where in the FPGA design these statistics are measured.
The smartnic platform implements reconfigurable ingress and egress port remapping, connections and redirecting. You can inspect and modify these configuration points using the “sw” subcommand.
Most of the sw subcommands take one or more port bindings as parameters. The port bindings are of the form:
<port>:<port-connector>
Where:
<port> is one of
<port-connector> is context dependent and is one of
sn-cli sw status
The in-port-rename subcommand allows you to remap the identity of a smartnic platform physical ingress port to any logical port as seen by the user logic. Once remapped (eg. from a->b), all following logic in the smartnic will perceive that the packet arrived on ingress port b even though it physically arrived on port a. This can be useful for test injection scenarios but would typically be set to a straight-through mapping in production.
sn-cli sw in-port-rename a:b
To reset this mapping so each port maps to its usual identity:
sn-cli sw in-port-rename cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
The in-port-connect subcommand allows you to connect a logical input port to different processing pipelines within the smartnic. This can be used to connect to a p4 program or to custom logic within the user application. It can also be used to shunt all packets to a blackhole or to bypass packets around the user application entirely.
sn-cli sw in-port-connect cmac0:app0 cmac1:app0 host0:bypass host1:bypass
The bypass-connect subcommand allows you to connect input ports directly to output ports as they pass through the bypass path (ie. not through the user application). This is useful for providing direct connectivity from host PCIe PFs to 100G CMAC interfaces for network testing.
sn-cli sw bypass-connect host0:cmac0 host1:cmac1 cmac0:host0 cmac1:host1
NOTE any packets that follow the bypass path will not be processed by the user’s p4 program
The app0-port-redirect and app1-port-redirect subcommands allow the user to override the forwarding decisions made by the user application and/or p4 program and redirect any given output port to a different output port. This can be useful during development/debugging and in test fixtures.
NOTE there are separate overrides for the app0 outputs and the app1 outputs.
sn-cli sw app0-port-redirect cmac0:host0 cmac1:host1
sn-cli sw app1-port-redirect cmac0:host0 cmac1:host1
To reset this mapping so each output ports maps to its usual destination:
sn-cli sw app0-port-redirect cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
sn-cli sw app1-port-redirect cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
The user’s p4 application embedded within the smartnic design may have configurable lookup tables which are used during the wire-speed execution of the packet processing pipeline. The sn-p4-cli tool provides subcommands to help you to manage the rules in all of the lookup tables defined in your p4 program.
All commands described below are expected to be executed within the smartnic-fw container environment. Use this command to enter the appropriate environment.
docker compose exec smartnic-fw bash
The sn-p4-cli tool will automatically look for an environment variable called SN_P4_CLI_SERVER which can be set to the hostname of the sn-p4-agent that will perform all of the requested actions on the real hardware. In the smartnic-fw container, this value will already be set for you.
The info subcommand is used to display the pipeline structure, including table names, match fields (and their types), action names and the list of parameters for each action. This information can be used to formulate new rule definitions for the other subcommands.
sn-p4-cli info
The table-insert subcommand allows you to insert a new rule into a specified table.
sn-p4-cli table-insert <table-name> <action-name> --match <match-expr> [--param <param-expr>] [--priority <prio-val>]
Where:
<table-name> is the name of the table to be operated on<action-name> is the action that you would like to activate when this rule matches<match-expr> is one or more match expressions which collectively define when this rule should match a given packet
--match option may be specified multiple times and all match-exprs will be concatenated<param-expr> is one or more parameter values which will be returned as a result when this rule matches a given packet
--param option entirely.<prio-val> is the priority to be used to resolve scenarios where multiple matches could occur
--priority option is required for tables with CAM/TCAM type matches (prefix/range/ternary)--priority option is prohibited for tables without CAM/TCAM type mathesNOTE: You can find details about your pipeline structure and valid names by running the info subcommand.
The table-update subcommand allows you to update the action and parameters for an existing rule within a table
sn-p4-cli table-update <table-name> <new-action-name> --match <match-expr> [--param <new-param-expr>]
Where:
<table-name> is the table containing the rule to be updated<new-action-name> is the new action that should be applied when this rule matches<match-expr> is the exact original <match-expr> used when the original rule was inserted<new-param-expr> is the set of new parameters to be returned when this rule matches
The clear-all and table-clear and table-delete subcommands allow you to remove rules from tables with varying precision.
Clear all rules from all tables in the pipeline.
sn-p4-cli clear-all`
Clear all rules from a single specified table.
sn-p4-cli table-clear <table-name>
Remove a specific rule from a specific table.
table-delete <table-name> --match <match-expr>
Using the the p4bm-apply subcommand, a list of pipeline modifications can be applied from a file. A subset of the full p4bm simulator file format is supported by the sn-p4-cli command.
sn-p4-cli p4bm-apply <filename>
Supported actions within the p4bm file are:
table_insert <table-name> <action-name> <match-expr> => <param-expr> [priority]
clear_all
table_clear <table-name>
All comment characters # and text following them up to the end of the line are ignored.
When we’re finished using the smartnic runtime environment, we can stop and remove our docker containers.
docker compose down -v
The sn-stack environment can be started in a mode where the FPGA can be controlled by a DPDK application. Running in this mode requires a few carefully ordered steps.
Broadly speaking, the steps required to bring up a DPDK application are as follows:
vfio-pci kernel driver to each FPGA PCIe physical function (PF)
-a $SN_PCIE_DEV.0 to allow control of one or more specific FPGA PCIe PFs-d librte_net_qdma.so to dynamically link the correct Userspace Polled-Mode Driver (PMD) for the smartnic QDMA enginevfio-pci driverhugepages driversn-cli tool to configure some of the low-level hardware components in the FPGA
In the examples below, we will be running the pktgen-dpdk application to control packet tx/rx via the FPGA’s PCIe physical functions. This can be very useful for injecting packets into a design for testing behaviour on real hardware.
For more information about DPDK in general, see:
For more information about the pktgen-dpdk application, see:
Before you bring up the sn-stack, please ensure that you have uncommented this line in your .env file
COMPOSE_PROFILES=smartnic-dpdk
If you changed this while the stack was already running, you’ll need to restart the stack with down/up.
First, you’ll need to start up the pktgen application to open the vfio-pci device for PF0 and PF1 and take the FPGA out of reset.
$ docker compose exec smartnic-dpdk bash
root@smartnic-dpdk:/# pktgen -a $SN_PCIE_DEV.0 -a $SN_PCIE_DEV.1 -l 4-8 -n 4 -d librte_net_qdma.so --file-prefix $SN_PCIE_DEV- -- -v -m [5:6].0 -m [7:8].1
Pktgen:/> help
NOTE: Leave this application running while doing the remaining setup steps. The setup steps below must be re-run after each time you restart the pktgen application since the FPGA gets reset between runs.
Open a separate shell window which you will use for doing the low-level smartnic platform configuration.
Configure the Queue mappings for host PF0 and PF1 interfaces and bring up the physical ethernet ports using the smartnic-fw container.
$ docker compose exec smartnic-fw bash
root@smartnic-fw:/# sn-cli qdma setqs 1 1
root@smartnic-fw:/# sn-cli qdma status
root@smartnic-fw:/# sn-cli cmac enable
root@smartnic-fw:/# sn-cli cmac status
Setting up the queue mappings tells the smartnic platform which QDMA queues to use for h2c and c2h packets. Enabling the CMACs allows Rx and Tx packets to flow (look for MAC ENABLED/PHY UP).
In this lesson, you will follow simple steps that allow you to use DPDK and pktgen to interact with the P4 program that you have loaded onto the FPGA.
Example of streaming packets out of an interface from a pcap file rather than generating the packets within the UI.
Note the -s <P>:file.pcap option where P refers to the port number to bind the pcap file to.
root@smartnic-dpdk:/# pktgen -a $SN_PCIE_DEV.0 -a $SN_PCIE_DEV.1 -l 4-8 -n 4 -d librte_net_qdma.so --file-prefix $SN_PCIE_DEV- -- -v -m [5:6].0 -m [7:8].1 -s 1:your_custom.pcap
Pktgen:/> port 1
Pktgen:/> page pcap
Pktgen:/> page main
Pktgen:/> start 1
Pktgen:/> stop 1
Pktgen:/> clr
Example of running a particular test case via a script rather than typing at the UI
cat <<_EOF > /tmp/test.pkt
clr
set 1 size 1400
set 1 count 1000000
enable 0 capture
start 1
disable 0 capture
_EOF
root@smartnic-dpdk:/# pktgen -a $SN_PCIE_DEV.0 -a SN_PCIE_DEV.1 -l 4-8 -n 4 -d librte_net_qdma.so --file-prefix $SN_PCIE_DEV- -- -v -m [5:6].0 -m [7:8].1 -f /tmp/test.pkt
If pktgen isn’t starting, please consider the following troubleshooting steps:
Ensure you are using the correct profile in your sn-stack/.env file and that you are starting pktgen with the right command. For a more detailed understanding of the command, please refer to the pktgen documentation provided earlier.
If pktgen is starting, but packets aren’t flowing as expected, you can check the packet path using the following command inside the smartnic-fw container:
sn-cli probe stats
If packets sent to/from the host aren’t achieving line rate (100Gbps per port), it could be due to QDMA queue allocation. You can attempt to allocate more QDMA queues per port by setting sn-cli qdma setqs to values higher than 1 1.
If packets are egressing to the wrong port (whether CMAC or PF), it might be due to the sn-cli configuration. For example, here’s a script that routes all egress packets to CMAC1:
#!/bin/bash
sn-cli dev version
sn-cli sw in-port-rename cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
sn-cli sw app0-port-redirect cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
sn-cli sw app1-port-redirect cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
sn-cli sw bypass-connect cmac0:cmac0 cmac1:cmac1 host0:host0 host1:host1
sn-cli sw in-port-connect cmac0:app0 cmac1:app0 host0:app0 host1:app0
sn-cli sw status
sn-cli qdma setqs 1 1
sn-cli cmac enable
sn-cli cmac status
These steps should help you troubleshoot issues related to the pktgen-dpdk application effectively.
This tutorial is built on the following software/respositories along with versions/commits:
None to date. If you face any issues, please contact mailto:cs595-f2023-group@iit.edu
Congratulations! You’ve successfully set up Xilinx Alveo FPGAs as SmartNICs using the ESnet framework. This powerful combination opens doors to accelerated network processing and customization. Feel free to explore further, experiment, and innovate with FPGA-based SmartNIC solutions.
For more information, visit the official ESnet repositories:
Happy networking!