1.. SPDX-License-Identifier: GPL-2.0 2 3==================================== 4HOWTO for the linux packet generator 5==================================== 6 7Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel 8or as a module. A module is preferred; modprobe pktgen if needed. Once 9running, pktgen creates a thread for each CPU with affinity to that CPU. 10Monitoring and controlling is done via /proc. It is easiest to select a 11suitable sample script and configure that. 12 13On a dual CPU:: 14 15 ps aux | grep pkt 16 root 129 0.3 0.0 0 0 ? SW 2003 523:20 [kpktgend_0] 17 root 130 0.3 0.0 0 0 ? SW 2003 509:50 [kpktgend_1] 18 19 20For monitoring and control pktgen creates:: 21 22 /proc/net/pktgen/pgctrl 23 /proc/net/pktgen/kpktgend_X 24 /proc/net/pktgen/ethX 25 26 27Tuning NIC for max performance 28============================== 29 30The default NIC settings are (likely) not tuned for pktgen's artificial 31overload type of benchmarking, as this could hurt the normal use-case. 32 33Specifically increasing the TX ring buffer in the NIC:: 34 35 # ethtool -G ethX tx 1024 36 37A larger TX ring can improve pktgen's performance, while it can hurt 38in the general case, 1) because the TX ring buffer might get larger 39than the CPU's L1/L2 cache, 2) because it allows more queueing in the 40NIC HW layer (which is bad for bufferbloat). 41 42One should hesitate to conclude that packets/descriptors in the HW 43TX ring cause delay. Drivers usually delay cleaning up the 44ring-buffers for various performance reasons, and packets stalling 45the TX ring might just be waiting for cleanup. 46 47This cleanup issue is specifically the case for the driver ixgbe 48(Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups, 49and the cleanup interval is affected by the ethtool --coalesce setting 50of parameter "rx-usecs". 51 52For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):: 53 54 # ethtool -C ethX rx-usecs 30 55 56 57Kernel threads 58============== 59Pktgen creates a thread for each CPU with affinity to that CPU. 60Which is controlled through procfile /proc/net/pktgen/kpktgend_X. 61 62Example: /proc/net/pktgen/kpktgend_0:: 63 64 Running: 65 Stopped: eth4@0 66 Result: OK: add_device=eth4@0 67 68Most important are the devices assigned to the thread. 69 70The two basic thread commands are: 71 72 * add_device DEVICE@NAME -- adds a single device 73 * rem_device_all -- remove all associated devices 74 75When adding a device to a thread, a corresponding procfile is created 76which is used for configuring this device. Thus, device names need to 77be unique. 78 79To support adding the same device to multiple threads, which is useful 80with multi queue NICs, the device naming scheme is extended with "@": 81device@something 82 83The part after "@" can be anything, but it is custom to use the thread 84number. 85 86Viewing devices 87=============== 88 89The Params section holds configured information. The Current section 90holds running statistics. The Result is printed after a run or after 91interruption. Example:: 92 93 /proc/net/pktgen/eth4@0 94 95 Params: count 100000 min_pkt_size: 60 max_pkt_size: 60 96 frags: 0 delay: 0 clone_skb: 64 ifname: eth4@0 97 flows: 0 flowlen: 0 98 queue_map_min: 0 queue_map_max: 0 99 dst_min: 192.168.81.2 dst_max: 100 src_min: src_max: 101 src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8 102 udp_src_min: 9 udp_src_max: 109 udp_dst_min: 9 udp_dst_max: 9 103 src_mac_count: 0 dst_mac_count: 0 104 Flags: UDPSRC_RND NO_TIMESTAMP QUEUE_MAP_CPU 105 Current: 106 pkts-sofar: 100000 errors: 0 107 started: 623913381008us stopped: 623913396439us idle: 25us 108 seq_num: 100001 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 109 cur_saddr: 192.168.8.3 cur_daddr: 192.168.81.2 110 cur_udp_dst: 9 cur_udp_src: 42 111 cur_queue_map: 0 112 flows: 0 113 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags) 114 6480562pps 3110Mb/sec (3110669760bps) errors: 0 115 116 117Configuring devices 118=================== 119This is done via the /proc interface, and most easily done via pgset 120as defined in the sample scripts. 121You need to specify PGDEV environment variable to use functions from sample 122scripts, i.e.:: 123 124 export PGDEV=/proc/net/pktgen/eth4@0 125 source samples/pktgen/functions.sh 126 127Examples:: 128 129 pg_ctrl start starts injection. 130 pg_ctrl stop aborts injection. Also, ^C aborts generator. 131 132 pgset "clone_skb 1" sets the number of copies of the same packet 133 pgset "clone_skb 0" use single SKB for all transmits 134 pgset "burst 8" uses xmit_more API to queue 8 copies of the same 135 packet and update HW tx queue tail pointer once. 136 "burst 1" is the default 137 pgset "pkt_size 9014" sets packet size to 9014 138 pgset "frags 5" packet will consist of 5 fragments 139 pgset "count 200000" sets number of packets to send, set to zero 140 for continuous sends until explicitly stopped. 141 142 pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds 143 144 pgset "dst 10.0.0.1" sets IP destination address 145 (BEWARE! This generator is very aggressive!) 146 147 pgset "dst_min 10.0.0.1" Same as dst 148 pgset "dst_max 10.0.0.254" Set the maximum destination IP. 149 pgset "src_min 10.0.0.1" Set the minimum (or only) source IP. 150 pgset "src_max 10.0.0.254" Set the maximum source IP. 151 pgset "dst6 fec0::1" IPV6 destination address 152 pgset "src6 fec0::2" IPV6 source address 153 pgset "dstmac 00:00:00:00:00:00" sets MAC destination address 154 pgset "srcmac 00:00:00:00:00:00" sets MAC source address 155 156 pgset "queue_map_min 0" Sets the min value of tx queue interval 157 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices 158 To select queue 1 of a given device, 159 use queue_map_min=1 and queue_map_max=1 160 161 pgset "src_mac_count 1" Sets the number of MACs we'll range through. 162 The 'minimum' MAC is what you set with srcmac. 163 164 pgset "dst_mac_count 1" Sets the number of MACs we'll range through. 165 The 'minimum' MAC is what you set with dstmac. 166 167 pgset "flag [name]" Set a flag to determine behaviour. Current flags 168 are: IPSRC_RND # IP source is random (between min/max) 169 IPDST_RND # IP destination is random 170 UDPSRC_RND, UDPDST_RND, 171 MACSRC_RND, MACDST_RND 172 TXSIZE_RND, IPV6, 173 MPLS_RND, VID_RND, SVID_RND 174 FLOW_SEQ, 175 QUEUE_MAP_RND # queue map random 176 QUEUE_MAP_CPU # queue map mirrors smp_processor_id() 177 UDPCSUM, 178 IPSEC # IPsec encapsulation (needs CONFIG_XFRM) 179 NODE_ALLOC # node specific memory allocation 180 NO_TIMESTAMP # disable timestamping 181 pgset 'flag ![name]' Clear a flag to determine behaviour. 182 Note that you might need to use single quote in 183 interactive mode, so that your shell wouldn't expand 184 the specified flag as a history command. 185 186 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet. 187 188 pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then 189 cycle through the port range. 190 191 pgset "udp_src_max 9" set UDP source port max. 192 pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then 193 cycle through the port range. 194 pgset "udp_dst_max 9" set UDP destination port max. 195 196 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example 197 outer label=16,middle label=32, 198 inner label=0 (IPv4 NULL)) Note that 199 there must be no spaces between the 200 arguments. Leading zeros are required. 201 Do not set the bottom of stack bit, 202 that's done automatically. If you do 203 set the bottom of stack bit, that 204 indicates that you want to randomly 205 generate that address and the flag 206 MPLS_RND will be turned on. You 207 can have any mix of random and fixed 208 labels in the label stack. 209 210 pgset "mpls 0" turn off mpls (or any invalid argument works too!) 211 212 pgset "vlan_id 77" set VLAN ID 0-4095 213 pgset "vlan_p 3" set priority bit 0-7 (default 0) 214 pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0) 215 216 pgset "svlan_id 22" set SVLAN ID 0-4095 217 pgset "svlan_p 3" set priority bit 0-7 (default 0) 218 pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0) 219 220 pgset "vlan_id 9999" > 4095 remove vlan and svlan tags 221 pgset "svlan 9999" > 4095 remove svlan tag 222 223 224 pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00) 225 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00) 226 227 pgset "rate 300M" set rate to 300 Mb/s 228 pgset "ratep 1000000" set rate to 1Mpps 229 230 pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb() 231 Works with "burst" but not with "clone_skb". 232 Default xmit_mode is "start_xmit". 233 234Sample scripts 235============== 236 237A collection of tutorial scripts and helpers for pktgen is in the 238samples/pktgen directory. The helper parameters.sh file support easy 239and consistent parameter parsing across the sample scripts. 240 241Usage example and help:: 242 243 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2 244 245Usage::: 246 247 ./pktgen_sample01_simple.sh [-vx] -i ethX 248 249 -i : ($DEV) output interface/device (required) 250 -s : ($PKT_SIZE) packet size 251 -d : ($DEST_IP) destination IP. CIDR (e.g. 198.18.0.0/15) is also allowed 252 -m : ($DST_MAC) destination MAC-addr 253 -p : ($DST_PORT) destination PORT range (e.g. 433-444) is also allowed 254 -t : ($THREADS) threads to start 255 -f : ($F_THREAD) index of first thread (zero indexed CPU number) 256 -c : ($SKB_CLONE) SKB clones send before alloc new SKB 257 -n : ($COUNT) num messages to send per thread, 0 means indefinitely 258 -b : ($BURST) HW level bursting of SKBs 259 -v : ($VERBOSE) verbose 260 -x : ($DEBUG) debug 261 -6 : ($IP6) IPv6 262 -w : ($DELAY) Tx Delay value (ns) 263 -a : ($APPEND) Script will not reset generator's state, but will append its config 264 265The global variables being set are also listed. E.g. the required 266interface/device parameter "-i" sets variable $DEV. Copy the 267pktgen_sampleXX scripts and modify them to fit your own needs. 268 269 270Interrupt affinity 271=================== 272Note that when adding devices to a specific CPU it is a good idea to 273also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound 274to the same CPU. This reduces cache bouncing when freeing skbs. 275 276Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue 277to the running threads CPU (directly from smp_processor_id()). 278 279Enable IPsec 280============ 281Default IPsec transformation with ESP encapsulation plus transport mode 282can be enabled by simply setting:: 283 284 pgset "flag IPSEC" 285 pgset "flows 1" 286 287To avoid breaking existing testbed scripts for using AH type and tunnel mode, 288you can use "pgset spi SPI_VALUE" to specify which transformation mode 289to employ. 290 291 292Current commands and configuration options 293========================================== 294 295**Pgcontrol commands**:: 296 297 start 298 stop 299 reset 300 301**Thread commands**:: 302 303 add_device 304 rem_device_all 305 306 307**Device commands**:: 308 309 count 310 clone_skb 311 burst 312 debug 313 314 frags 315 delay 316 317 src_mac_count 318 dst_mac_count 319 320 pkt_size 321 min_pkt_size 322 max_pkt_size 323 324 queue_map_min 325 queue_map_max 326 skb_priority 327 328 tos (ipv4) 329 traffic_class (ipv6) 330 331 mpls 332 333 udp_src_min 334 udp_src_max 335 336 udp_dst_min 337 udp_dst_max 338 339 node 340 341 flag 342 IPSRC_RND 343 IPDST_RND 344 UDPSRC_RND 345 UDPDST_RND 346 MACSRC_RND 347 MACDST_RND 348 TXSIZE_RND 349 IPV6 350 MPLS_RND 351 VID_RND 352 SVID_RND 353 FLOW_SEQ 354 QUEUE_MAP_RND 355 QUEUE_MAP_CPU 356 UDPCSUM 357 IPSEC 358 NODE_ALLOC 359 NO_TIMESTAMP 360 361 spi (ipsec) 362 363 dst_min 364 dst_max 365 366 src_min 367 src_max 368 369 dst_mac 370 src_mac 371 372 clear_counters 373 374 src6 375 dst6 376 dst6_max 377 dst6_min 378 379 flows 380 flowlen 381 382 rate 383 ratep 384 385 xmit_mode <start_xmit|netif_receive> 386 387 vlan_cfi 388 vlan_id 389 vlan_p 390 391 svlan_cfi 392 svlan_id 393 svlan_p 394 395 396References: 397 398- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/ 399- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/ 400 401Paper from Linux-Kongress in Erlangen 2004. 402- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf 403 404Thanks to: 405 406Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek 407Stephen Hemminger, Andi Kleen, Dave Miller and many others. 408 409 410Good luck with the linux net-development. 411