1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License (the "License"). 6 * You may not use this file except in compliance with the License. 7 * 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9 * or http://www.opensolaris.org/os/licensing. 10 * See the License for the specific language governing permissions 11 * and limitations under the License. 12 * 13 * When distributing Covered Code, include this CDDL HEADER in each 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15 * If applicable, add the following below this CDDL HEADER, with the 16 * fields enclosed by brackets "[]" replaced with your own identifying 17 * information: Portions Copyright [yyyy] [name of copyright owner] 18 * 19 * CDDL HEADER END 20 */ 21 /* 22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved. 23 * Use is subject to license terms. 24 * Copyright 2018 Joyent, Inc. 25 * Copyright 2013 Nexenta Systems, Inc. All rights reserved. 26 * Copyright 2025 Oxide Computer Company 27 */ 28 29 /* 30 * MAC data path 31 * 32 * The MAC data path is concerned with the flow of traffic from mac clients -- 33 * DLS, IP, etc. -- to various GLDv3 device drivers -- e1000g, vnic, aggr, 34 * ixgbe, etc. -- and from the GLDv3 device drivers back to clients. 35 * 36 * ----------- 37 * Terminology 38 * ----------- 39 * 40 * MAC uses a lot of different, but related terms that are associated with the 41 * design and structure of the data path. Before we cover other aspects, first 42 * let's review the terminology that MAC uses. 43 * 44 * MAC 45 * 46 * This driver. It interfaces with device drivers and provides abstractions 47 * that the rest of the system consumes. All data links -- things managed 48 * with dladm(8), are accessed through MAC. 49 * 50 * GLDv3 DEVICE DRIVER 51 * 52 * A GLDv3 device driver refers to a driver, both for pseudo-devices and 53 * real devices, which implement the GLDv3 driver API. Common examples of 54 * these are igb and ixgbe, which are drivers for various Intel networking 55 * cards. These devices may or may not have various features, such as 56 * hardware rings and checksum offloading. For MAC, a GLDv3 device is the 57 * final point for the transmission of a packet and the starting point for 58 * the receipt of a packet. 59 * 60 * FLOWS 61 * 62 * At a high level, a flow refers to a series of packets that are related. 63 * Often times the term is used in the context of TCP to indicate a unique 64 * TCP connection and the traffic over it. However, a flow can exist at 65 * other levels of the system as well. MAC has a notion of a default flow 66 * which is used for all unicast traffic addressed to the address of a MAC 67 * device. For example, when a VNIC is created, a default flow is created 68 * for the VNIC's MAC address. In addition, flows are created for broadcast 69 * groups and a user may create a flow with flowadm(8). 70 * 71 * CLASSIFICATION 72 * 73 * Classification refers to the notion of identifying an incoming frame 74 * based on its destination address and optionally its source addresses and 75 * doing different processing based on that information. Classification can 76 * be done in both hardware and software. In general, we usually only 77 * classify based on the layer two destination, eg. for Ethernet, the 78 * destination MAC address. 79 * 80 * The system also will do classification based on layer three and layer 81 * four properties. This is used to support things like flowadm(8), which 82 * allows setting QoS and other properties on a per-flow basis. 83 * 84 * RING 85 * 86 * Conceptually, a ring represents a series of framed messages, often in a 87 * contiguous chunk of memory that acts as a circular buffer. Rings come in 88 * a couple of forms. Generally they are either a hardware construct (hw 89 * ring) or they are a software construct (sw ring) maintained by MAC. 90 * 91 * HW RING 92 * 93 * A hardware ring is a set of resources provided by a GLDv3 device driver 94 * (even if it is a pseudo-device). A hardware ring comes in two different 95 * forms: receive (rx) rings and transmit (tx) rings. An rx hw ring is 96 * something that has a unique DMA (direct memory access) region and 97 * generally supports some form of classification (though it isn't always 98 * used), as well as a means of generating an interrupt specific to that 99 * ring. For example, the device may generate a specific MSI-X for a PCI 100 * express device. A tx ring is similar, except that it is dedicated to 101 * transmission. It may also be a vector for enabling features such as VLAN 102 * tagging and large transmit offloading. It usually has its own dedicated 103 * interrupts for transmit being completed. 104 * 105 * SW RING 106 * 107 * A software ring is a construction of MAC. It represents the same thing 108 * that a hardware ring generally does, a collection of frames. However, 109 * instead of being in a contiguous ring of memory, they're instead linked 110 * by using the mblk_t's b_next pointer. Each frame may itself be multiple 111 * mblk_t's linked together by the b_cont pointer. A software ring always 112 * represents a collection of classified packets; however, it varies as to 113 * whether it uses only layer two information, or a combination of that and 114 * additional layer three and layer four data. 115 * 116 * FANOUT 117 * 118 * Fanout is the idea of spreading out the load of processing frames based 119 * on the source and destination information contained in the layer two, 120 * three, and four headers, such that the data can then be processed in 121 * parallel using multiple hardware threads. 122 * 123 * A fanout algorithm hashes the headers and uses that to place different 124 * flows into a bucket. The most important thing is that packets that are 125 * in the same flow end up in the same bucket. If they do not, performance 126 * can be adversely affected. Consider the case of TCP. TCP severely 127 * penalizes a connection if the data arrives out of order. If a given flow 128 * is processed on different CPUs, then the data will appear out of order, 129 * hence the invariant that fanout always hash a given flow to the same 130 * bucket and thus get processed on the same CPU. 131 * 132 * RECEIVE SIDE SCALING (RSS) 133 * 134 * 135 * Receive side scaling is a term that isn't common in illumos, but is used 136 * by vendors and was popularized by Microsoft. It refers to the idea of 137 * spreading the incoming receive load out across multiple interrupts which 138 * can be directed to different CPUs. This allows a device to leverage 139 * hardware rings even when it doesn't support hardware classification. The 140 * hardware uses an algorithm to perform fanout that ensures the flow 141 * invariant is maintained. 142 * 143 * SOFT RING SET 144 * 145 * A soft ring set, commonly abbreviated SRS, is a collection of rings and 146 * is used for both transmitting and receiving. It is maintained in the 147 * structure mac_soft_ring_set_t. A soft ring set is usually associated 148 * with flows, and coordinates both the use of hardware and software rings. 149 * Because the use of hardware rings can change as devices such as VNICs 150 * come and go, we always ensure that the set has software classification 151 * rules that correspond to the hardware classification rules from rings. 152 * 153 * Soft ring sets are also used for the enforcement of various QoS 154 * properties. For example, if a bandwidth limit has been placed on a 155 * specific flow or device, then that will be enforced by the soft ring 156 * set. 157 * 158 * SERVICE ATTACHMENT POINT (SAP) 159 * 160 * The service attachment point is a DLPI (Data Link Provider Interface) 161 * concept; however, it comes up quite often in MAC. Most MAC devices speak 162 * a protocol that has some notion of different channels or message type 163 * identifiers. For example, Ethernet defines an EtherType which is a part 164 * of the Ethernet header and defines the particular protocol of the data 165 * payload. If the EtherType is set to 0x0800, then it defines that the 166 * contents of that Ethernet frame is IPv4 traffic. For Ethernet, the 167 * EtherType is the SAP. 168 * 169 * In DLPI, a given consumer attaches to a specific SAP. In illumos, the ip 170 * and arp drivers attach to the EtherTypes for IPv4, IPv6, and ARP. Using 171 * libdlpi(3LIB) user software can attach to arbitrary SAPs. With the 172 * exception of 802.1Q VLAN tagged traffic, MAC itself does not directly 173 * consume the SAP; however, it uses that information as part of hashing 174 * and it may be used as part of the construction of flows. 175 * 176 * PRIMARY MAC CLIENT 177 * 178 * The primary mac client refers to a mac client whose unicast address 179 * matches the address of the device itself. For example, if the system has 180 * instance of the e1000g driver such as e1000g0, e1000g1, etc., the 181 * primary mac client is the one named after the device itself. VNICs that 182 * are created on top of such devices are not the primary client. 183 * 184 * TRANSMIT DESCRIPTORS 185 * 186 * Transmit descriptors are a resource that most GLDv3 device drivers have. 187 * Generally, a GLDv3 device driver takes a frame that's meant to be output 188 * and puts a copy of it into a region of memory. Each region of memory 189 * usually has an associated descriptor that the device uses to manage 190 * properties of the frames. Devices have a limited number of such 191 * descriptors. They get reclaimed once the device finishes putting the 192 * frame on the wire. 193 * 194 * If the driver runs out of transmit descriptors, for example, the OS is 195 * generating more frames than it can put on the wire, then it will return 196 * them back to the MAC layer. 197 * 198 * --------------------------------- 199 * Rings, Classification, and Fanout 200 * --------------------------------- 201 * 202 * The heart of MAC is made up of rings, and not those that Elven-kings wear. 203 * When receiving a packet, MAC breaks the work into two different, though 204 * interrelated phases. The first phase is generally classification and then the 205 * second phase is generally fanout. When a frame comes in from a GLDv3 Device, 206 * MAC needs to determine where that frame should be delivered. If it's a 207 * unicast frame (say a normal TCP/IP packet), then it will be delivered to a 208 * single MAC client; however, if it's a broadcast or multicast frame, then MAC 209 * may need to deliver it to multiple MAC clients. 210 * 211 * On transmit, classification isn't quite as important, but may still be used. 212 * Unlike with the receive path, the classification is not used to determine 213 * devices that should transmit something, but rather is used for special 214 * properties of a flow, eg. bandwidth limits for a given IP address, device, or 215 * connection. 216 * 217 * MAC employs a software classifier and leverages hardware classification as 218 * well. The software classifier can leverage the full layer two information, 219 * source, destination, VLAN, and SAP. If the SAP indicates that IP traffic is 220 * being sent, it can classify based on the IP header, and finally, it also 221 * knows how to classify based on the local and remote ports of TCP, UDP, and 222 * SCTP. 223 * 224 * Hardware classifiers vary in capability. Generally all hardware classifiers 225 * provide the capability to classify based on the destination MAC address. Some 226 * hardware has additional filters built in for performing more in-depth 227 * classification; however, it often has much more limited resources for these 228 * activities as compared to the layer two destination address classification. 229 * 230 * The modus operandi in MAC is to always ensure that we have software-based 231 * capabilities and rules in place and then to supplement that with hardware 232 * resources when available. In general, simple layer two classification is 233 * sufficient and nothing else is used, unless a specific flow is created with 234 * tools such as flowadm(8) or bandwidth limits are set on a device with 235 * dladm(8). 236 * 237 * RINGS AND GROUPS 238 * 239 * To get into how rings and classification play together, it's first important 240 * to understand how hardware devices commonly associate rings and allow them to 241 * be programmed. Recall that a hardware ring should be thought of as a DMA 242 * buffer and an interrupt resource. Rings are then collected into groups. A 243 * group itself has a series of classification rules. One or more MAC addresses 244 * are assigned to a group. 245 * 246 * Hardware devices vary in terms of what capabilities they provide. Sometimes 247 * they allow for a dynamic assignment of rings to a group and sometimes they 248 * have a static assignment of rings to a group. For example, the ixgbe driver 249 * has a static assignment of rings to groups such that every group has exactly 250 * one ring and the number of groups is equal to the number of rings. 251 * 252 * Classification and receive side scaling both come into play with how a device 253 * advertises itself to MAC and how MAC uses it. If a device supports layer two 254 * classification of frames, then MAC will assign MAC addresses to a group as a 255 * form of primary classification. If a single MAC address is assigned to a 256 * group, a common case, then MAC will consider packets that come in from rings 257 * on that group to be fully classified and will not need to do any software 258 * classification unless a specific flow has been created. 259 * 260 * If a device supports receive side scaling, then it may advertise or support 261 * groups with multiple rings. In those cases, then receive side scaling will 262 * come into play and MAC will use that as a means of fanning out received 263 * frames across multiple CPUs. This can also be combined with groups that 264 * support layer two classification. 265 * 266 * If a device supports dynamic assignments of rings to groups, then MAC will 267 * change around the way that rings are assigned to various groups as devices 268 * come and go from the system. For example, when a VNIC is created, a new flow 269 * will be created for the VNIC's MAC address. If a hardware ring is available, 270 * MAC may opt to reassign it from one group to another. 271 * 272 * ASSIGNMENT OF HARDWARE RINGS 273 * 274 * This is a bit of a complicated subject that varies depending on the device, 275 * the use of aggregations, the special nature of the primary mac client. This 276 * section deserves being fleshed out. 277 * 278 * FANOUT 279 * 280 * illumos uses fanout to help spread out the incoming processing load of chains 281 * of frames away from a single CPU. If a device supports receive side scaling, 282 * then that provides an initial form of fanout; however, what we're concerned 283 * with all happens after the context of a given set of frames being classified 284 * to a soft ring set. 285 * 286 * After frames reach a soft ring set and account for any potential bandwidth 287 * related accounting, they may be fanned out based on one of the following 288 * three modes: 289 * 290 * o No Fanout 291 * o Protocol level fanout 292 * o Full software ring protocol fanout 293 * 294 * MAC makes the determination as to which of these modes a given soft ring set 295 * obtains based on parameters such as whether or not it's the primary mac 296 * client, whether it's on a 10 GbE or faster device, user controlled dladm(8) 297 * properties, and the nature of the hardware and the resources that it has. 298 * 299 * When there is no fanout, MAC does not create any soft rings for a device and 300 * the device has frames delivered directly to the MAC client. 301 * 302 * Otherwise, all fanout is performed by software. MAC divides incoming frames 303 * into one of five buckets -- IPv4 TCP traffic, IPv4 UDP traffic, IPv6 TCP 304 * traffic, IPv6 UDP traffic, and everything else. Regardless of the type of 305 * fanout, these five categories of buckets are always used. 306 * 307 * The difference between protocol level fanout and full software ring protocol 308 * fanout is the number of software rings that end up getting created. The 309 * system always uses the same number of software rings per protocol bucket. So 310 * in the first case when we're just doing protocol level fanout, we just create 311 * one software ring each for IPv4 TCP traffic, IPv4 UDP traffic, IPv6 TCP 312 * traffic, IPv6 UDP traffic, and everything else. 313 * 314 * In the case where we do full software ring protocol fanout, we generally use 315 * mac_compute_soft_ring_count() to determine the number of rings. There are 316 * other combinations of properties and devices that may send us down other 317 * paths, but this is a common starting point. If it's a non-bandwidth enforced 318 * device and we're on at least a 10 GbE link, then we'll use eight soft rings 319 * per protocol bucket as a starting point. See mac_compute_soft_ring_count() 320 * for more information on the total number. 321 * 322 * For each of these rings, we create a mac_soft_ring_t and an associated worker 323 * thread. Particularly when doing full software ring protocol fanout, we bind 324 * each of the worker threads to individual CPUs. 325 * 326 * The other advantage of these software rings is that it allows upper layers to 327 * optionally poll on them. For example, TCP can leverage an squeue to poll on 328 * the software ring, see squeue.c for more information. 329 * 330 * DLS BYPASS 331 * 332 * DLS is the data link services module. It interfaces with DLPI, which is the 333 * primary way that other parts of the system such as IP interface with the MAC 334 * layer. While DLS is traditionally a STREAMS-based interface, it allows for 335 * certain modules such as IP to negotiate various more modern interfaces to be 336 * used, which are useful for higher performance and allow it to use direct 337 * function calls to DLS instead of using STREAMS. 338 * 339 * When we have TCP or UDP software rings, then traffic on those rings is 340 * eligible for what we call the dls bypass. In those cases, rather than going 341 * out mac_rx_deliver() to DLS, DLS instead registers them to go directly via 342 * the direct callback registered with DLS, generally ip_input(). 343 * 344 * HARDWARE RING POLLING 345 * 346 * GLDv3 devices with hardware rings generally deliver chains of messages 347 * (mblk_t chain) during the context of a single interrupt. However, interrupts 348 * are not the only way that these devices may be used. As part of implementing 349 * ring support, a GLDv3 device driver must have a way to disable the generation 350 * of that interrupt and allow for the operating system to poll on that ring. 351 * 352 * To implement this, every soft ring set has a worker thread and a polling 353 * thread. If a sufficient packet rate comes into the system, MAC will 'blank' 354 * (disable) interrupts on that specific ring and the polling thread will start 355 * consuming packets from the hardware device and deliver them to the soft ring 356 * set, where the worker thread will take over. 357 * 358 * Once the rate of packet intake drops down below a certain threshold, then 359 * polling on the hardware ring will be quiesced and interrupts will be 360 * re-enabled for the given ring. This effectively allows the system to shift 361 * how it handles a ring based on its load. At high packet rates, polling on the 362 * device as opposed to relying on interrupts can actually reduce overall system 363 * load due to the minimization of interrupt activity. 364 * 365 * Note the importance of each ring having its own interrupt source. The whole 366 * idea here is that we do not disable interrupts on the device as a whole, but 367 * rather each ring can be independently toggled. 368 * 369 * USE OF WORKER THREADS 370 * 371 * Both the soft ring set and individual soft rings have a worker thread 372 * associated with them that may be bound to a specific CPU in the system. Any 373 * such assignment will get reassessed as part of dynamic reconfiguration events 374 * in the system such as the onlining and offlining of CPUs and the creation of 375 * CPU partitions. 376 * 377 * In many cases, while in an interrupt, we try to deliver a frame all the way 378 * through the stack in the context of the interrupt itself. However, if the 379 * amount of queued frames has exceeded a threshold, then we instead defer to 380 * the worker thread to do this work and signal it. This is particularly useful 381 * when you have the soft ring set delivering frames into multiple software 382 * rings. If it was only delivering frames into a single software ring then 383 * there'd be no need to have another thread take over. However, if it's 384 * delivering chains of frames to multiple rings, then it's worthwhile to have 385 * the worker for the software ring take over so that the different software 386 * rings can be processed in parallel. 387 * 388 * In a similar fashion to the hardware polling thread, if we don't have a 389 * backlog or there's nothing to do, then the worker thread will go back to 390 * sleep and frames can be delivered all the way from an interrupt. This 391 * behavior is useful as it's designed to minimize latency and the default 392 * disposition of MAC is to optimize for latency. 393 * 394 * MAINTAINING CHAINS 395 * 396 * Another useful idea that MAC uses is to try and maintain frames in chains for 397 * as long as possible. The idea is that all of MAC can handle chains of frames 398 * structured as a series of mblk_t structures linked with the b_next pointer. 399 * When performing software classification and software fanout, MAC does not 400 * simply determine the destination and send the frame along. Instead, in the 401 * case of classification, it tries to maintain a chain for as long as possible 402 * before passing it along and performing additional processing. 403 * 404 * In the case of fanout, MAC first determines what the target software ring is 405 * for every frame in the original chain and constructs a new chain for each 406 * target. MAC then delivers the new chain to each software ring in succession. 407 * 408 * The whole rationale for doing this is that we want to try and maintain the 409 * pipe as much as possible and deliver as many frames through the stack at once 410 * that we can, rather than just pushing a single frame through. This can often 411 * help bring down latency and allows MAC to get a better sense of the overall 412 * activity in the system and properly engage worker threads. 413 * 414 * -------------------- 415 * Bandwidth Management 416 * -------------------- 417 * 418 * Bandwidth management is something that's built into the soft ring set itself. 419 * When bandwidth limits are placed on a flow, a corresponding soft ring set is 420 * toggled into bandwidth mode. This changes how we transmit and receive the 421 * frames in question. 422 * 423 * Bandwidth management is done on a per-tick basis. We translate the user's 424 * requested bandwidth from a quantity per-second into a quantity per-tick. MAC 425 * cannot process a frame across more than one tick, thus it sets a lower bound 426 * for the bandwidth cap to be a single MTU. This also means that when 427 * hires ticks are enabled (hz is set to 1000), that the minimum amount of 428 * bandwidth is higher, because the number of ticks has increased and MAC has to 429 * go from accepting 100 packets / sec to 1000 / sec. 430 * 431 * The bandwidth counter is reset by either the soft ring set's worker thread or 432 * a thread that is doing an inline transmit or receive if they discover that 433 * the current tick is in the future from the recorded tick. 434 * 435 * Whenever we're receiving or transmitting data, we end up leaving most of the 436 * work to the soft ring set's worker thread. This forces data inserted into the 437 * soft ring set to be effectively serialized and allows us to exhume bandwidth 438 * at a reasonable rate. If there is nothing in the soft ring set at the moment 439 * and the set has available bandwidth, then it may processed inline. 440 * Otherwise, the worker is responsible for taking care of the soft ring set. 441 * 442 * --------------------- 443 * The Receive Data Path 444 * --------------------- 445 * 446 * The following series of ASCII art images breaks apart the way that a frame 447 * comes in and is processed in MAC. 448 * 449 * Part 1 -- Initial frame receipt, SRS classification 450 * 451 * Here, a frame is received by a GLDv3 driver, generally in the context of an 452 * interrupt, and it ends up in mac_rx_common(). A driver calls either mac_rx or 453 * mac_rx_ring, depending on whether or not it supports rings and can identify 454 * the interrupt as having come from a specific ring. Here we determine whether 455 * or not it's fully classified and perform software classification as 456 * appropriate. From here, everything always ends up going to either entry [A] 457 * or entry [B] based on whether or not they have subflow processing needed. We 458 * leave via fanout or delivery. 459 * 460 * +===========+ 461 * v hardware v 462 * v interrupt v 463 * +===========+ 464 * | 465 * * . . appropriate 466 * | upcall made 467 * | by GLDv3 driver . . always 468 * | . 469 * +--------+ | +----------+ . +---------------+ 470 * | GLDv3 | +---->| mac_rx |-----*--->| mac_rx_common | 471 * | Driver |-->--+ +----------+ +---------------+ 472 * +--------+ | ^ | 473 * | | ^ v 474 * ^ | * . . always +----------------------+ 475 * | | | | mac_promisc_dispatch | 476 * | | +-------------+ +----------------------+ 477 * | +--->| mac_rx_ring | | 478 * | +-------------+ * . . hw classified 479 * | v or single flow? 480 * | | 481 * | +--------++--------------+ 482 * | | | * hw class, 483 * | | * hw classified | subflows 484 * | no hw class and . * | or single | exist 485 * | subflows | | flow | 486 * | | v v 487 * | | +-----------+ +-----------+ 488 * | | | goto | | goto | 489 * | | | entry [A] | | entry [B] | 490 * | | +-----------+ +-----------+ 491 * | v ^ 492 * | +-------------+ | 493 * | | mac_rx_flow | * SRS and flow found, 494 * | +-------------+ | call flow cb 495 * | | +------+ 496 * | v | 497 * v +==========+ +-----------------+ 498 * | v For each v--->| mac_rx_classify | 499 * +----------+ v mblk_t v +-----------------+ 500 * | srs | +==========+ 501 * | pollling | 502 * | thread |->------------------------------------------+ 503 * +----------+ | 504 * v . inline 505 * +--------------------+ +----------+ +---------+ . 506 * [A]---->| mac_rx_srs_process |-->| check bw |-->| enqueue |--*---------+ 507 * +--------------------+ | limits | | frames | | 508 * ^ +----------+ | to SRS | | 509 * | +---------+ | 510 * | send chain +--------+ | | 511 * * when clasified | signal | * BW limits, | 512 * | flow changes | srs |<---+ loopback, | 513 * | | worker | stack too | 514 * | +--------+ deep | 515 * +-----------------+ +--------+ | 516 * | mac_flow_lookup | | srs | +---------------------+ | 517 * +-----------------+ | worker |---->| mac_rx_srs_drain |<---+ 518 * ^ | thread | | mac_rx_srs_drain_bw | 519 * | +--------+ +---------------------+ 520 * | | 521 * +----------------------------+ * software rings 522 * [B]-->| mac_rx_srs_subflow_process | | for fanout? 523 * +----------------------------+ | 524 * +----------+-----------+ 525 * | | 526 * v v 527 * +--------+ +--------+ 528 * | goto | | goto | 529 * | Part 2 | | Part 3 | 530 * +--------+ +--------+ 531 * 532 * Part 2 -- Fanout 533 * 534 * This part is concerned with using software fanout to assign frames to 535 * software rings and then deliver them to MAC clients or allow those rings to 536 * be polled upon. While there are two different primary fanout entry points, 537 * mac_rx_fanout and mac_rx_proto_fanout, they behave in similar ways, and aside 538 * from some of the individual hashing techniques used, most of the general 539 * flow is the same. 540 * 541 * +--------+ +-------------------+ 542 * | From |---+--------->| mac_rx_srs_fanout |----+ 543 * | Part 1 | | +-------------------+ | +=================+ 544 * +--------+ | | v for each mblk_t v 545 * * . . protocol only +--->v assign to new v 546 * | fanout | v chain based on v 547 * | | v hash % nrings v 548 * | +-------------------------+ | +=================+ 549 * +--->| mac_rx_srs_proto_fanout |----+ | 550 * +-------------------------+ | 551 * v 552 * +------------+ +--------------------------+ +================+ 553 * | enqueue in |<---| mac_rx_soft_ring_process |<------v for each chain v 554 * | soft ring | +--------------------------+ +================+ 555 * +------------+ 556 * | +-----------+ 557 * * soft ring set | soft ring | 558 * | empty and no | worker | 559 * | worker? | thread | 560 * | +-----------+ 561 * +------*----------------+ | 562 * | . | v 563 * No . * . Yes | +------------------------+ 564 * | +----<--| mac_rx_soft_ring_drain | 565 * | | +------------------------+ 566 * v | 567 * +-----------+ v 568 * | signal | +---------------+ 569 * | soft ring | | Deliver chain | 570 * | worker | | goto Part 3 | 571 * +-----------+ +---------------+ 572 * 573 * 574 * Part 3 -- Packet Delivery 575 * 576 * Here, we go through and deliver the mblk_t chain directly to a given 577 * processing function. In a lot of cases this is mac_rx_deliver(). In the case 578 * of DLS bypass being used, then instead we end up going ahead and deliver it 579 * to the direct callback registered with DLS, generally ip_input. 580 * 581 * 582 * +---------+ +----------------+ +------------------+ 583 * | From |---+------->| mac_rx_deliver |--->| Off to DLS, or | 584 * | Parts 1 | | +----------------+ | other MAC client | 585 * | and 2 | * DLS bypass +------------------+ 586 * +---------+ | enabled +----------+ +-------------+ 587 * +---------->| ip_input |--->| To IP | 588 * +----------+ | and beyond! | 589 * +-------------+ 590 * 591 * ---------------------- 592 * The Transmit Data Path 593 * ---------------------- 594 * 595 * Before we go into the images, it's worth talking about a problem that is a 596 * bit different from the receive data path. GLDv3 device drivers have a finite 597 * amount of transmit descriptors. When they run out, they return unused frames 598 * back to MAC. MAC, at this point has several options about what it will do, 599 * which vary based upon the settings that the client uses. 600 * 601 * When a device runs out of descriptors, the next thing that MAC does is 602 * enqueue them off of the soft ring set or a software ring, depending on the 603 * configuration of the soft ring set. MAC will enqueue up to a high watermark 604 * of mblk_t chains, at which point it will indicate flow control back to the 605 * client. Once this condition is reached, any mblk_t chains that were not 606 * enqueued will be returned to the caller and they will have to decide what to 607 * do with them. There are various flags that control this behavior that a 608 * client may pass, which are discussed below. 609 * 610 * When this condition is hit, MAC also returns a cookie to the client in 611 * addition to unconsumed frames. Clients can poll on that cookie and register a 612 * callback with MAC to be notified when they are no longer subject to flow 613 * control, at which point they may continue to call mac_tx(). This flow control 614 * actually manages to work itself all the way up the stack, back through dls, 615 * to ip, through the various protocols, and to sockfs. 616 * 617 * While the behavior described above is the default, this behavior can be 618 * modified. There are two alternate modes, described below, which are 619 * controlled with flags. 620 * 621 * DROP MODE 622 * 623 * This mode is controlled by having the client pass the MAC_DROP_ON_NO_DESC 624 * flag. When this is passed, if a device driver runs out of transmit 625 * descriptors, then the MAC layer will drop any unsent traffic. The client in 626 * this case will never have any frames returned to it. 627 * 628 * DON'T ENQUEUE 629 * 630 * This mode is controlled by having the client pass the MAC_TX_NO_ENQUEUE flag. 631 * If the MAC_DROP_ON_NO_DESC flag is also passed, it takes precedence. In this 632 * mode, when we hit a case where a driver runs out of transmit descriptors, 633 * then instead of enqueuing packets in a soft ring set or software ring, we 634 * instead return the mblk_t chain back to the caller and immediately put the 635 * soft ring set into flow control mode. 636 * 637 * The following series of ASCII art images describe the transmit data path that 638 * MAC clients enter into based on calling into mac_tx(). A soft ring set has a 639 * transmission function associated with it. There are seven possible 640 * transmission modes, some of which share function entry points. The one that a 641 * soft ring set gets depends on properties such as whether there are 642 * transmission rings for fanout, whether the device involves aggregations, 643 * whether any bandwidth limits exist, etc. 644 * 645 * 646 * Part 1 -- Initial checks 647 * 648 * * . called by 649 * | MAC clients 650 * v . . No 651 * +--------+ +-----------+ . +-------------------+ +====================+ 652 * | mac_tx |->| device |-*-->| mac_protect_check |->v Is this the simple v 653 * +--------+ | quiesced? | +-------------------+ v case? See [1] v 654 * +-----------+ | +====================+ 655 * * . Yes * failed | 656 * v | frames | 657 * +--------------+ | +-------+---------+ 658 * | freemsgchain |<---------+ Yes . * No . * 659 * +--------------+ v v 660 * +-----------+ +--------+ 661 * | goto | | goto | 662 * | Part 2 | | SRS TX | 663 * | Entry [A] | | func | 664 * +-----------+ +--------+ 665 * | | 666 * | v 667 * | +--------+ 668 * +---------->| return | 669 * | cookie | 670 * +--------+ 671 * 672 * [1] The simple case refers to the SRS being configured with the 673 * SRS_TX_DEFAULT transmission mode, having a single mblk_t (not a chain), their 674 * being only a single active client, and not having a backlog in the srs. 675 * 676 * 677 * Part 2 -- The SRS transmission functions 678 * 679 * This part is a bit more complicated. The different transmission paths often 680 * leverage one another. In this case, we'll draw out the more common ones 681 * before the parts that depend upon them. Here, we're going to start with the 682 * workings of mac_tx_send() a common function that most of the others end up 683 * calling. 684 * 685 * +-------------+ 686 * | mac_tx_send | 687 * +-------------+ 688 * | 689 * v 690 * +=============+ +==============+ 691 * v more than v--->v check v 692 * v one client? v v VLAN and add v 693 * +=============+ v VLAN tags v 694 * | +==============+ 695 * | | 696 * +------------------+ 697 * | 698 * | [A] 699 * v | 700 * +============+ . No v 701 * v more than v . +==========+ +--------------------------+ 702 * v one active v-*---->v for each v---->| mac_promisc_dispatch_one |---+ 703 * v client? v v mblk_t v +--------------------------+ | 704 * +============+ +==========+ ^ | 705 * | | +==========+ | 706 * * . Yes | v hardware v<-------+ 707 * v +------------+ v rings? v 708 * +==========+ | +==========+ 709 * v for each v No . . . * | 710 * v mblk_t v specific | | 711 * +==========+ flow | +-----+-----+ 712 * | | | | 713 * v | v v 714 * +-----------------+ | +-------+ +---------+ 715 * | mac_tx_classify |------------+ | GLDv3 | | GLDv3 | 716 * +-----------------+ |TX func| | ring tx | 717 * | +-------+ | func | 718 * * Specific flow, generally | +---------+ 719 * | bcast, mcast, loopback | | 720 * v +-----+-----+ 721 * +==========+ +---------+ | 722 * v valid L2 v--*--->| freemsg | v 723 * v header v . No +---------+ +-------------------+ 724 * +==========+ | return unconsumed | 725 * * . Yes | frames to the | 726 * v | caller | 727 * +===========+ +-------------------+ 728 * v braodcast v +----------------+ ^ 729 * v flow? v--*-->| mac_bcast_send |------------------+ 730 * +===========+ . +----------------+ | 731 * | . . Yes | 732 * No . * v 733 * | +---------------------+ +---------------+ +----------+ 734 * +->|mac_promisc_dispatch |->| mac_fix_cksum |->| flow | 735 * +---------------------+ +---------------+ | callback | 736 * +----------+ 737 * 738 * 739 * In addition, many but not all of the routines, all rely on 740 * mac_tx_softring_process as an entry point. 741 * 742 * 743 * . No . No 744 * +--------------------------+ +========+ . +===========+ . +-------------+ 745 * | mac_tx_soft_ring_process |-->v worker v-*->v out of tx v-*->| goto | 746 * +--------------------------+ v only? v v descr.? v | mac_tx_send | 747 * +========+ +===========+ +-------------+ 748 * Yes . * * . Yes | 749 * . No v | v 750 * v=========+ . +===========+ . Yes | Yes . +==========+ 751 * v apppend v<--*----------v out of tx v-*-------+---------*--v returned v 752 * v mblk_t v v descr.? v | v frames? v 753 * v chain v +===========+ | +==========+ 754 * +=========+ | *. No 755 * | | v 756 * v v +------------+ 757 * +===================+ +----------------------+ | done | 758 * v worker scheduled? v | mac_tx_sring_enqueue | | processing | 759 * v Out of tx descr? v +----------------------+ +------------+ 760 * +===================+ | 761 * | | . Yes v 762 * * Yes * No . +============+ 763 * | v +-*---------v drop on no v 764 * | +========+ v v TX desc? v 765 * | v wake v +----------+ +============+ 766 * | v worker v | mac_pkt_ | * . No 767 * | +========+ | drop | | . Yes . No 768 * | | +----------+ v . . 769 * | | v ^ +===============+ . +========+ . 770 * +--+--------+---------+ | v Don't enqueue v-*->v ring v-*----+ 771 * | | v Set? v v empty? v | 772 * | +---------------+ +===============+ +========+ | 773 * | | | | | 774 * | | +-------------------+ | | 775 * | *. Yes | +---------+ | 776 * | | v v v 777 * | | +===========+ +========+ +--------------+ 778 * | +<-v At hiwat? v v append v | return | 779 * | +===========+ v mblk_t v | mblk_t chain | 780 * | * No v chain v | and flow | 781 * | v +========+ | control | 782 * | +=========+ | | cookie | 783 * | v append v v +--------------+ 784 * | v mblk_t v +========+ 785 * | v chain v v wake v +------------+ 786 * | +=========+ v worker v-->| done | 787 * | | +========+ | processing | 788 * | v .. Yes +------------+ 789 * | +=========+ . +========+ 790 * | v first v--*-->v wake v 791 * | v append? v v worker v 792 * | +=========+ +========+ 793 * | | | 794 * | No . * | 795 * | v | 796 * | +--------------+ | 797 * +------>| Return | | 798 * | flow control |<------------+ 799 * | cookie | 800 * +--------------+ 801 * 802 * 803 * The remaining images are all specific to each of the different transmission 804 * modes. 805 * 806 * SRS TX DEFAULT 807 * 808 * [ From Part 1 ] 809 * | 810 * v 811 * +-------------------------+ 812 * | mac_tx_single_ring_mode | 813 * +-------------------------+ 814 * | 815 * | . Yes 816 * v . 817 * +==========+ . +============+ 818 * v SRS v-*->v Try to v---->---------------------+ 819 * v backlog? v v enqueue in v | 820 * +==========+ v SRS v-->------+ * . . Queue too 821 * | +============+ * don't enqueue | deep or 822 * * . No ^ | | flag or at | drop flag 823 * | | v | hiwat, | 824 * v | | | return +---------+ 825 * +-------------+ | | | cookie | freemsg | 826 * | goto |-*-----+ | | +---------+ 827 * | mac_tx_send | . returned | | | 828 * +-------------+ mblk_t | | | 829 * | | | | 830 * | | | | 831 * * . . all mblk_t * queued, | | 832 * v consumed | may return | | 833 * +-------------+ | tx cookie | | 834 * | SRS TX func |<------------+------------+----------------+ 835 * | completed | 836 * +-------------+ 837 * 838 * SRS_TX_SERIALIZE 839 * 840 * +------------------------+ 841 * | mac_tx_serializer_mode | 842 * +------------------------+ 843 * | 844 * | . No 845 * v . 846 * +============+ . +============+ +-------------+ +============+ 847 * v srs being v-*->v set SRS v--->| goto |-->v remove SRS v 848 * v processed? v v proc flags v | mac_tx_send | v proc flag v 849 * +============+ +============+ +-------------+ +============+ 850 * | | 851 * * Yes | 852 * v . No v 853 * +--------------------+ . +==========+ 854 * | mac_tx_srs_enqueue | +------------------------*-----<--v returned v 855 * +--------------------+ | v frames? v 856 * | | . Yes +==========+ 857 * | | . | 858 * | | . +=========+ v 859 * v +-<-*-v queued v +--------------------+ 860 * +-------------+ | v frames? v<----| mac_tx_srs_enqueue | 861 * | SRS TX func | | +=========+ +--------------------+ 862 * | completed, |<------+ * . Yes 863 * | may return | | v 864 * | cookie | | +========+ 865 * +-------------+ +-<---v wake v 866 * v worker v 867 * +========+ 868 * 869 * 870 * SRS_TX_FANOUT 871 * 872 * . Yes 873 * +--------------------+ +=============+ . +--------------------------+ 874 * | mac_tx_fanout_mode |--->v Have fanout v-*-->| goto | 875 * +--------------------+ v hint? v | mac_rx_soft_ring_process | 876 * +=============+ +--------------------------+ 877 * * . No | 878 * v ^ 879 * +===========+ | 880 * +--->v for each v +===============+ 881 * | v mblk_t v v pick softring v 882 * same * +===========+ v from hash v 883 * hash | | +===============+ 884 * | v | 885 * | +--------------+ | 886 * +---| mac_pkt_hash |--->*------------+ 887 * +--------------+ . different 888 * hash or 889 * done proc. 890 * SRS_TX_AGGR chain 891 * 892 * +------------------+ +================================+ 893 * | mac_tx_aggr_mode |--->v Use aggr capab function to v 894 * +------------------+ v find appropriate tx ring. v 895 * v Applies hash based on aggr v 896 * v policy, see mac_tx_aggr_mode() v 897 * +================================+ 898 * | 899 * v 900 * +-------------------------------+ 901 * | goto | 902 * | mac_rx_srs_soft_ring_process | 903 * +-------------------------------+ 904 * 905 * 906 * SRS_TX_BW, SRS_TX_BW_FANOUT, SRS_TX_BW_AGGR 907 * 908 * Note, all three of these tx functions start from the same place -- 909 * mac_tx_bw_mode(). 910 * 911 * +----------------+ 912 * | mac_tx_bw_mode | 913 * +----------------+ 914 * | 915 * v . No . No . Yes 916 * +==============+ . +============+ . +=============+ . +=========+ 917 * v Out of BW? v--*->v SRS empty? v--*->v reset BW v-*->v Bump BW v 918 * +==============+ +============+ v tick count? v v Usage v 919 * | | +=============+ +=========+ 920 * | +---------+ | | 921 * | | +--------------------+ | 922 * | | | +----------------------+ 923 * v | v v 924 * +===============+ | +==========+ +==========+ +------------------+ 925 * v Don't enqueue v | v set bw v v Is aggr? v--*-->| goto | 926 * v flag set? v | v enforced v +==========+ . | mac_tx_aggr_mode |-+ 927 * +===============+ | +==========+ | . +------------------+ | 928 * | Yes .* | | No . * . | 929 * | | | | | . Yes | 930 * * . No | | v | | 931 * | +---------+ | +========+ v +======+ | 932 * | | freemsg | | v append v +============+ . Yes v pick v | 933 * | +---------+ | v mblk_t v v Is fanout? v--*---->v ring v | 934 * | | | v chain v +============+ +======+ | 935 * +------+ | +========+ | | | 936 * v | | v v | 937 * +---------+ | v +-------------+ +--------------------+ | 938 * | return | | +========+ | goto | | goto | | 939 * | flow | | v wakeup v | mac_tx_send | | mac_tx_fanout_mode | | 940 * | control | | v worker v +-------------+ +--------------------+ | 941 * | cookie | | +========+ | | | 942 * +---------+ | | | +------+------+ 943 * | v | | 944 * | +---------+ | v 945 * | | return | +============+ +------------+ 946 * | | flow | v unconsumed v-------+ | done | 947 * | | control | v frames? v | | processing | 948 * | | cookie | +============+ | +------------+ 949 * | +---------+ | | 950 * | Yes * | 951 * | | | 952 * | +===========+ | 953 * | v subtract v | 954 * | v unused bw v | 955 * | +===========+ | 956 * | | | 957 * | v | 958 * | +--------------------+ | 959 * +------------->| mac_tx_srs_enqueue | | 960 * +--------------------+ | 961 * | | 962 * | | 963 * +------------+ | 964 * | return fc | | 965 * | cookie and |<------+ 966 * | mblk_t | 967 * +------------+ 968 */ 969 970 #include <sys/types.h> 971 #include <sys/callb.h> 972 #include <sys/pattr.h> 973 #include <sys/sdt.h> 974 #include <sys/strsubr.h> 975 #include <sys/strsun.h> 976 #include <sys/vlan.h> 977 #include <sys/stack.h> 978 #include <sys/archsystm.h> 979 #include <inet/ipsec_impl.h> 980 #include <inet/ip_impl.h> 981 #include <inet/sadb.h> 982 #include <inet/ipsecesp.h> 983 #include <inet/ipsecah.h> 984 #include <inet/ip6.h> 985 986 #include <sys/mac_impl.h> 987 #include <sys/mac_client_impl.h> 988 #include <sys/mac_client_priv.h> 989 #include <sys/mac_soft_ring.h> 990 #include <sys/mac_flow_impl.h> 991 992 static mac_tx_cookie_t mac_tx_single_ring_mode(mac_soft_ring_set_t *, mblk_t *, 993 uintptr_t, uint16_t, mblk_t **); 994 static mac_tx_cookie_t mac_tx_serializer_mode(mac_soft_ring_set_t *, mblk_t *, 995 uintptr_t, uint16_t, mblk_t **); 996 static mac_tx_cookie_t mac_tx_fanout_mode(mac_soft_ring_set_t *, mblk_t *, 997 uintptr_t, uint16_t, mblk_t **); 998 static mac_tx_cookie_t mac_tx_bw_mode(mac_soft_ring_set_t *, mblk_t *, 999 uintptr_t, uint16_t, mblk_t **); 1000 static mac_tx_cookie_t mac_tx_aggr_mode(mac_soft_ring_set_t *, mblk_t *, 1001 uintptr_t, uint16_t, mblk_t **); 1002 1003 typedef struct mac_tx_mode_s { 1004 mac_tx_srs_mode_t mac_tx_mode; 1005 mac_tx_func_t mac_tx_func; 1006 } mac_tx_mode_t; 1007 1008 /* 1009 * There are seven modes of operation on the Tx side. These modes get set 1010 * in mac_tx_srs_setup(). Except for the experimental TX_SERIALIZE mode, 1011 * none of the other modes are user configurable. They get selected by 1012 * the system depending upon whether the link (or flow) has multiple Tx 1013 * rings or a bandwidth configured, or if the link is an aggr, etc. 1014 * 1015 * When the Tx SRS is operating in aggr mode (st_mode) or if there are 1016 * multiple Tx rings owned by Tx SRS, then each Tx ring (pseudo or 1017 * otherwise) will have a soft ring associated with it. These soft rings 1018 * are stored in srs_tx_soft_rings[] array. 1019 * 1020 * Additionally in the case of aggr, there is the st_soft_rings[] array 1021 * in the mac_srs_tx_t structure. This array is used to store the same 1022 * set of soft rings that are present in srs_tx_soft_rings[] array but 1023 * in a different manner. The soft ring associated with the pseudo Tx 1024 * ring is saved at mr_index (of the pseudo ring) in st_soft_rings[] 1025 * array. This helps in quickly getting the soft ring associated with the 1026 * Tx ring when aggr_find_tx_ring() returns the pseudo Tx ring that is to 1027 * be used for transmit. 1028 */ 1029 mac_tx_mode_t mac_tx_mode_list[] = { 1030 {SRS_TX_DEFAULT, mac_tx_single_ring_mode}, 1031 {SRS_TX_SERIALIZE, mac_tx_serializer_mode}, 1032 {SRS_TX_FANOUT, mac_tx_fanout_mode}, 1033 {SRS_TX_BW, mac_tx_bw_mode}, 1034 {SRS_TX_BW_FANOUT, mac_tx_bw_mode}, 1035 {SRS_TX_AGGR, mac_tx_aggr_mode}, 1036 {SRS_TX_BW_AGGR, mac_tx_bw_mode} 1037 }; 1038 1039 /* 1040 * Soft Ring Set (SRS) - The Run time code that deals with 1041 * dynamic polling from the hardware, bandwidth enforcement, 1042 * fanout etc. 1043 * 1044 * We try to use H/W classification on NIC and assign traffic for 1045 * a MAC address to a particular Rx ring or ring group. There is a 1046 * 1-1 mapping between a SRS and a Rx ring. The SRS dynamically 1047 * switches the underlying Rx ring between interrupt and 1048 * polling mode and enforces any specified B/W control. 1049 * 1050 * There is always a SRS created and tied to each H/W and S/W rule. 1051 * Whenever we create a H/W rule, we always add the the same rule to 1052 * S/W classifier and tie a SRS to it. 1053 * 1054 * In case a B/W control is specified, it is broken into bytes 1055 * per ticks and as soon as the quota for a tick is exhausted, 1056 * the underlying Rx ring is forced into poll mode for remainder of 1057 * the tick. The SRS poll thread only polls for bytes that are 1058 * allowed to come in the SRS. We typically let 4x the configured 1059 * B/W worth of packets to come in the SRS (to prevent unnecessary 1060 * drops due to bursts) but only process the specified amount. 1061 * 1062 * A MAC client (e.g. a VNIC or aggr) can have 1 or more 1063 * Rx rings (and corresponding SRSs) assigned to it. The SRS 1064 * in turn can have softrings to do protocol level fanout or 1065 * softrings to do S/W based fanout or both. In case the NIC 1066 * has no Rx rings, we do S/W classification to respective SRS. 1067 * The S/W classification rule is always setup and ready. This 1068 * allows the MAC layer to reassign Rx rings whenever needed 1069 * but packets still continue to flow via the default path and 1070 * getting S/W classified to correct SRS. 1071 * 1072 * The SRS's are used on both Tx and Rx side. They use the same 1073 * data structure but the processing routines have slightly different 1074 * semantics due to the fact that Rx side needs to do dynamic 1075 * polling etc. 1076 * 1077 * Dynamic Polling Notes 1078 * ===================== 1079 * 1080 * Each Soft ring set is capable of switching its Rx ring between 1081 * interrupt and poll mode and actively 'polls' for packets in 1082 * poll mode. If the SRS is implementing a B/W limit, it makes 1083 * sure that only Max allowed packets are pulled in poll mode 1084 * and goes to poll mode as soon as B/W limit is exceeded. As 1085 * such, there are no overheads to implement B/W limits. 1086 * 1087 * In poll mode, its better to keep the pipeline going where the 1088 * SRS worker thread keeps processing packets and poll thread 1089 * keeps bringing more packets (specially if they get to run 1090 * on different CPUs). This also prevents the overheads associated 1091 * by excessive signalling (on NUMA machines, this can be 1092 * pretty devastating). The exception is latency optimized case 1093 * where worker thread does no work and interrupt and poll thread 1094 * are allowed to do their own drain. 1095 * 1096 * We use the following policy to control Dynamic Polling: 1097 * 1) We switch to poll mode anytime the processing 1098 * thread causes a backlog to build up in SRS and 1099 * its associated Soft Rings (sr_poll_pkt_cnt > 0). 1100 * 2) As long as the backlog stays under the low water 1101 * mark (sr_lowat), we poll the H/W for more packets. 1102 * 3) If the backlog (sr_poll_pkt_cnt) exceeds low 1103 * water mark, we stay in poll mode but don't poll 1104 * the H/W for more packets. 1105 * 4) Anytime in polling mode, if we poll the H/W for 1106 * packets and find nothing plus we have an existing 1107 * backlog (sr_poll_pkt_cnt > 0), we stay in polling 1108 * mode but don't poll the H/W for packets anymore 1109 * (let the polling thread go to sleep). 1110 * 5) Once the backlog is relived (packets are processed) 1111 * we reenable polling (by signalling the poll thread) 1112 * only when the backlog dips below sr_poll_thres. 1113 * 6) sr_hiwat is used exclusively when we are not 1114 * polling capable and is used to decide when to 1115 * drop packets so the SRS queue length doesn't grow 1116 * infinitely. 1117 * 1118 * NOTE: Also see the block level comment on top of mac_soft_ring.c 1119 */ 1120 1121 /* 1122 * mac_latency_optimize 1123 * 1124 * Controls whether the poll thread can process the packets inline 1125 * or let the SRS worker thread do the processing. This applies if 1126 * the SRS was not being processed. For latency sensitive traffic, 1127 * this needs to be true to allow inline processing. For throughput 1128 * under load, this should be false. 1129 * 1130 * This (and other similar) tunable should be rolled into a link 1131 * or flow specific workload hint that can be set using dladm 1132 * linkprop (instead of multiple such tunables). 1133 */ 1134 boolean_t mac_latency_optimize = B_TRUE; 1135 1136 /* 1137 * MAC_RX_SRS_ENQUEUE_CHAIN and MAC_TX_SRS_ENQUEUE_CHAIN 1138 * 1139 * queue a mp or chain in soft ring set and increment the 1140 * local count (srs_count) for the SRS and the shared counter 1141 * (srs_poll_pkt_cnt - shared between SRS and its soft rings 1142 * to track the total unprocessed packets for polling to work 1143 * correctly). 1144 * 1145 * The size (total bytes queued) counters are incremented only 1146 * if we are doing B/W control. 1147 */ 1148 #define MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1149 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1150 if ((mac_srs)->srs_last != NULL) \ 1151 (mac_srs)->srs_last->b_next = (head); \ 1152 else \ 1153 (mac_srs)->srs_first = (head); \ 1154 (mac_srs)->srs_last = (tail); \ 1155 (mac_srs)->srs_count += count; \ 1156 } 1157 1158 #define MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1159 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \ 1160 \ 1161 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \ 1162 srs_rx->sr_poll_pkt_cnt += count; \ 1163 ASSERT(srs_rx->sr_poll_pkt_cnt > 0); \ 1164 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \ 1165 (mac_srs)->srs_size += (sz); \ 1166 mutex_enter(&(mac_srs)->srs_bw->mac_bw_lock); \ 1167 (mac_srs)->srs_bw->mac_bw_sz += (sz); \ 1168 mutex_exit(&(mac_srs)->srs_bw->mac_bw_lock); \ 1169 } \ 1170 } 1171 1172 #define MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1173 mac_srs->srs_state |= SRS_ENQUEUED; \ 1174 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \ 1175 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \ 1176 (mac_srs)->srs_size += (sz); \ 1177 (mac_srs)->srs_bw->mac_bw_sz += (sz); \ 1178 } \ 1179 } 1180 1181 /* 1182 * Turn polling on routines 1183 */ 1184 #define MAC_SRS_POLLING_ON(mac_srs) { \ 1185 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1186 if (((mac_srs)->srs_state & \ 1187 (SRS_POLLING_CAPAB|SRS_POLLING)) == SRS_POLLING_CAPAB) { \ 1188 (mac_srs)->srs_state |= SRS_POLLING; \ 1189 (void) mac_hwring_disable_intr((mac_ring_handle_t) \ 1190 (mac_srs)->srs_ring); \ 1191 (mac_srs)->srs_rx.sr_poll_on++; \ 1192 } \ 1193 } 1194 1195 #define MAC_SRS_WORKER_POLLING_ON(mac_srs) { \ 1196 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1197 if (((mac_srs)->srs_state & \ 1198 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_POLLING)) == \ 1199 (SRS_POLLING_CAPAB|SRS_WORKER)) { \ 1200 (mac_srs)->srs_state |= SRS_POLLING; \ 1201 (void) mac_hwring_disable_intr((mac_ring_handle_t) \ 1202 (mac_srs)->srs_ring); \ 1203 (mac_srs)->srs_rx.sr_worker_poll_on++; \ 1204 } \ 1205 } 1206 1207 /* 1208 * MAC_SRS_POLL_RING 1209 * 1210 * Signal the SRS poll thread to poll the underlying H/W ring 1211 * provided it wasn't already polling (SRS_GET_PKTS was set). 1212 * 1213 * Poll thread gets to run only from mac_rx_srs_drain() and only 1214 * if the drain was being done by the worker thread. 1215 */ 1216 #define MAC_SRS_POLL_RING(mac_srs) { \ 1217 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \ 1218 \ 1219 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1220 srs_rx->sr_poll_thr_sig++; \ 1221 if (((mac_srs)->srs_state & \ 1222 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_GET_PKTS)) == \ 1223 (SRS_WORKER|SRS_POLLING_CAPAB)) { \ 1224 (mac_srs)->srs_state |= SRS_GET_PKTS; \ 1225 cv_signal(&(mac_srs)->srs_cv); \ 1226 } else { \ 1227 srs_rx->sr_poll_thr_busy++; \ 1228 } \ 1229 } 1230 1231 /* 1232 * MAC_SRS_CHECK_BW_CONTROL 1233 * 1234 * Check to see if next tick has started so we can reset the 1235 * SRS_BW_ENFORCED flag and allow more packets to come in the 1236 * system. 1237 */ 1238 #define MAC_SRS_CHECK_BW_CONTROL(mac_srs) { \ 1239 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1240 ASSERT(((mac_srs)->srs_type & SRST_TX) || \ 1241 MUTEX_HELD(&(mac_srs)->srs_bw->mac_bw_lock)); \ 1242 clock_t now = ddi_get_lbolt(); \ 1243 if ((mac_srs)->srs_bw->mac_bw_curr_time != now) { \ 1244 (mac_srs)->srs_bw->mac_bw_curr_time = now; \ 1245 (mac_srs)->srs_bw->mac_bw_used = 0; \ 1246 if ((mac_srs)->srs_bw->mac_bw_state & SRS_BW_ENFORCED) \ 1247 (mac_srs)->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; \ 1248 } \ 1249 } 1250 1251 /* 1252 * MAC_SRS_WORKER_WAKEUP 1253 * 1254 * Wake up the SRS worker thread to process the queue as long as 1255 * no one else is processing the queue. If we are optimizing for 1256 * latency, we wake up the worker thread immediately or else we 1257 * wait mac_srs_worker_wakeup_ticks before worker thread gets 1258 * woken up. 1259 */ 1260 int mac_srs_worker_wakeup_ticks = 0; 1261 #define MAC_SRS_WORKER_WAKEUP(mac_srs) { \ 1262 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1263 if (!((mac_srs)->srs_state & SRS_PROC) && \ 1264 (mac_srs)->srs_tid == NULL) { \ 1265 if (((mac_srs)->srs_state & SRS_LATENCY_OPT) || \ 1266 (mac_srs_worker_wakeup_ticks == 0)) \ 1267 cv_signal(&(mac_srs)->srs_async); \ 1268 else \ 1269 (mac_srs)->srs_tid = \ 1270 timeout(mac_srs_fire, (mac_srs), \ 1271 mac_srs_worker_wakeup_ticks); \ 1272 } \ 1273 } 1274 1275 #define TX_BANDWIDTH_MODE(mac_srs) \ 1276 ((mac_srs)->srs_tx.st_mode == SRS_TX_BW || \ 1277 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_FANOUT || \ 1278 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_AGGR) 1279 1280 #define TX_SRS_TO_SOFT_RING(mac_srs, head, hint) { \ 1281 if (tx_mode == SRS_TX_BW_FANOUT) \ 1282 (void) mac_tx_fanout_mode(mac_srs, head, hint, 0, NULL);\ 1283 else \ 1284 (void) mac_tx_aggr_mode(mac_srs, head, hint, 0, NULL); \ 1285 } 1286 1287 /* 1288 * MAC_TX_SRS_BLOCK 1289 * 1290 * Always called from mac_tx_srs_drain() function. SRS_TX_BLOCKED 1291 * will be set only if srs_tx_woken_up is FALSE. If 1292 * srs_tx_woken_up is TRUE, it indicates that the wakeup arrived 1293 * before we grabbed srs_lock to set SRS_TX_BLOCKED. We need to 1294 * attempt to transmit again and not setting SRS_TX_BLOCKED does 1295 * that. 1296 */ 1297 #define MAC_TX_SRS_BLOCK(srs, mp) { \ 1298 ASSERT(MUTEX_HELD(&(srs)->srs_lock)); \ 1299 if ((srs)->srs_tx.st_woken_up) { \ 1300 (srs)->srs_tx.st_woken_up = B_FALSE; \ 1301 } else { \ 1302 ASSERT(!((srs)->srs_state & SRS_TX_BLOCKED)); \ 1303 (srs)->srs_state |= SRS_TX_BLOCKED; \ 1304 (srs)->srs_tx.st_stat.mts_blockcnt++; \ 1305 } \ 1306 } 1307 1308 /* 1309 * MAC_TX_SRS_TEST_HIWAT 1310 * 1311 * Called before queueing a packet onto Tx SRS to test and set 1312 * SRS_TX_HIWAT if srs_count exceeds srs_tx_hiwat. 1313 */ 1314 #define MAC_TX_SRS_TEST_HIWAT(srs, mp, tail, cnt, sz, cookie) { \ 1315 boolean_t enqueue = 1; \ 1316 \ 1317 if ((srs)->srs_count > (srs)->srs_tx.st_hiwat) { \ 1318 /* \ 1319 * flow-controlled. Store srs in cookie so that it \ 1320 * can be returned as mac_tx_cookie_t to client \ 1321 */ \ 1322 (srs)->srs_state |= SRS_TX_HIWAT; \ 1323 cookie = (mac_tx_cookie_t)srs; \ 1324 (srs)->srs_tx.st_hiwat_cnt++; \ 1325 if ((srs)->srs_count > (srs)->srs_tx.st_max_q_cnt) { \ 1326 /* increment freed stats */ \ 1327 (srs)->srs_tx.st_stat.mts_sdrops += cnt; \ 1328 /* \ 1329 * b_prev may be set to the fanout hint \ 1330 * hence can't use freemsg directly \ 1331 */ \ 1332 mac_drop_chain(mp_chain, "SRS Tx max queue"); \ 1333 DTRACE_PROBE1(tx_queued_hiwat, \ 1334 mac_soft_ring_set_t *, srs); \ 1335 enqueue = 0; \ 1336 } \ 1337 } \ 1338 if (enqueue) \ 1339 MAC_TX_SRS_ENQUEUE_CHAIN(srs, mp, tail, cnt, sz); \ 1340 } 1341 1342 /* Some utility macros */ 1343 #define MAC_SRS_BW_LOCK(srs) \ 1344 if (!(srs->srs_type & SRST_TX)) \ 1345 mutex_enter(&srs->srs_bw->mac_bw_lock); 1346 1347 #define MAC_SRS_BW_UNLOCK(srs) \ 1348 if (!(srs->srs_type & SRST_TX)) \ 1349 mutex_exit(&srs->srs_bw->mac_bw_lock); 1350 1351 #define MAC_TX_SRS_DROP_MESSAGE(srs, chain, cookie, s) { \ 1352 mac_drop_chain((chain), (s)); \ 1353 /* increment freed stats */ \ 1354 (srs)->srs_tx.st_stat.mts_sdrops++; \ 1355 (cookie) = (mac_tx_cookie_t)(srs); \ 1356 } 1357 1358 #define MAC_TX_SET_NO_ENQUEUE(srs, mp_chain, ret_mp, cookie) { \ 1359 mac_srs->srs_state |= SRS_TX_WAKEUP_CLIENT; \ 1360 cookie = (mac_tx_cookie_t)srs; \ 1361 *ret_mp = mp_chain; \ 1362 } 1363 1364 /* 1365 * Threshold used in receive-side processing to determine if handling 1366 * can occur in situ (in the interrupt thread) or if it should be left to a 1367 * worker thread. Note that the constant used to make this determination is 1368 * not entirely made-up, and is a result of some emprical validation. That 1369 * said, the constant is left as a global variable to allow it to be 1370 * dynamically tuned in the field if and as needed. 1371 */ 1372 uintptr_t mac_rx_srs_stack_needed = 14336; 1373 uint_t mac_rx_srs_stack_toodeep; 1374 1375 #ifndef STACK_GROWTH_DOWN 1376 #error Downward stack growth assumed. 1377 #endif 1378 1379 /* 1380 * Drop the rx packet and advance to the next one in the chain. 1381 */ 1382 static void 1383 mac_rx_drop_pkt(mac_soft_ring_set_t *srs, mblk_t *mp) 1384 { 1385 mac_srs_rx_t *srs_rx = &srs->srs_rx; 1386 1387 ASSERT(mp->b_next == NULL); 1388 mutex_enter(&srs->srs_lock); 1389 MAC_UPDATE_SRS_COUNT_LOCKED(srs, 1); 1390 MAC_UPDATE_SRS_SIZE_LOCKED(srs, msgdsize(mp)); 1391 mutex_exit(&srs->srs_lock); 1392 1393 srs_rx->sr_stat.mrs_sdrops++; 1394 freemsg(mp); 1395 } 1396 1397 /* DATAPATH RUNTIME ROUTINES */ 1398 1399 /* 1400 * mac_srs_fire 1401 * 1402 * Timer callback routine for waking up the SRS worker thread. 1403 */ 1404 static void 1405 mac_srs_fire(void *arg) 1406 { 1407 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)arg; 1408 1409 mutex_enter(&mac_srs->srs_lock); 1410 if (mac_srs->srs_tid == NULL) { 1411 mutex_exit(&mac_srs->srs_lock); 1412 return; 1413 } 1414 1415 mac_srs->srs_tid = NULL; 1416 if (!(mac_srs->srs_state & SRS_PROC)) 1417 cv_signal(&mac_srs->srs_async); 1418 1419 mutex_exit(&mac_srs->srs_lock); 1420 } 1421 1422 /* 1423 * 'hint' is fanout_hint (type of uint64_t) which is given by the TCP/IP stack, 1424 * and it is used on the TX path. 1425 */ 1426 #define HASH_HINT(hint) \ 1427 ((hint) ^ ((hint) >> 24) ^ ((hint) >> 16) ^ ((hint) >> 8)) 1428 1429 1430 /* 1431 * hash based on the src address, dst address and the port information. 1432 */ 1433 #define HASH_ADDR(src, dst, ports) \ 1434 (ntohl((src) + (dst)) ^ ((ports) >> 24) ^ ((ports) >> 16) ^ \ 1435 ((ports) >> 8) ^ (ports)) 1436 1437 /* 1438 * Uniform distribution hash for IPv6 4-tuple. 1439 */ 1440 #define HASH_ADDR6(src, dst, ports) \ 1441 ((src.s6_addr32[0] ^ src.s6_addr32[1] ^ \ 1442 src.s6_addr32[2] ^ src.s6_addr32[3]) ^ \ 1443 (dst.s6_addr32[0] ^ dst.s6_addr32[1] ^ \ 1444 dst.s6_addr32[2] ^ dst.s6_addr32[3]) ^ \ 1445 ((ports) >> 24) ^ ((ports) >> 16) ^ \ 1446 ((ports) >> 8) ^ (ports)) 1447 1448 #define COMPUTE_INDEX(key, sz) (key % sz) 1449 1450 #define FANOUT_ENQUEUE_MP(head, tail, cnt, bw_ctl, sz, sz0, mp) { \ 1451 if ((tail) != NULL) { \ 1452 ASSERT((tail)->b_next == NULL); \ 1453 (tail)->b_next = (mp); \ 1454 } else { \ 1455 ASSERT((head) == NULL); \ 1456 (head) = (mp); \ 1457 } \ 1458 (tail) = (mp); \ 1459 (cnt)++; \ 1460 if ((bw_ctl)) \ 1461 (sz) += (sz0); \ 1462 } 1463 1464 #define MAC_FANOUT_DEFAULT 0 1465 #define MAC_FANOUT_RND_ROBIN 1 1466 int mac_fanout_type = MAC_FANOUT_DEFAULT; 1467 1468 #define MAX_SR_TYPES 5 1469 /* fanout types for port based hashing */ 1470 typedef enum pkt_type { 1471 V4_TCP = 0, 1472 V4_UDP, 1473 V6_TCP, 1474 V6_UDP, 1475 OTH, 1476 UNDEF 1477 } pkt_type_t; 1478 1479 /* 1480 * Pair of local and remote ports in the transport header 1481 */ 1482 #define PORTS_SIZE 4 1483 1484 /* 1485 * This routine delivers packets destined for an SRS into one of the 1486 * protocol soft rings. 1487 * 1488 * Given a chain of packets we need to split it up into multiple sub 1489 * chains: TCP, UDP or OTH soft ring. Instead of entering the soft 1490 * ring one packet at a time, we want to enter it in the form of a 1491 * chain otherwise we get this start/stop behaviour where the worker 1492 * thread goes to sleep and then next packet comes in forcing it to 1493 * wake up. 1494 */ 1495 static void 1496 mac_rx_srs_proto_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head) 1497 { 1498 mblk_t *headmp[MAX_SR_TYPES] = { 0 }; 1499 mblk_t *tailmp[MAX_SR_TYPES] = { 0 }; 1500 int cnt[MAX_SR_TYPES] = { 0 }; 1501 size_t sz[MAX_SR_TYPES] = { 0 }; 1502 mac_client_impl_t *mcip = mac_srs->srs_mcip; 1503 1504 const boolean_t is_ether = 1505 (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER); 1506 const boolean_t bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0); 1507 1508 /* 1509 * If we don't have a Rx ring, S/W classification would have done 1510 * its job and its a packet meant for us. If we were polling on 1511 * the default ring (i.e. there was a ring assigned to this SRS), 1512 * then we need to make sure that the mac address really belongs 1513 * to us. 1514 */ 1515 const boolean_t hw_classified = mac_srs->srs_ring != NULL && 1516 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER; 1517 1518 /* 1519 * Some clients, such as non-ethernet, need DLS processing in 1520 * the Rx path. Such clients clear the SRST_DLS_BYPASS flag. 1521 * DLS bypass may also be disabled via the 1522 * MCIS_RX_BYPASS_DISABLE flag. 1523 */ 1524 const boolean_t dls_bypass = 1525 ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) && 1526 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0); 1527 1528 /* 1529 * We have a chain from SRS that we need to split across the 1530 * soft rings. The squeues for the TCP and IPv4 SAPs use their 1531 * own soft rings to allow polling from the squeue. The rest of 1532 * the packets are delivered on the OTH soft ring which cannot 1533 * be polled. 1534 */ 1535 while (head != NULL) { 1536 mac_ether_offload_info_t meoi = { 0 }; 1537 uint8_t ether_addr[ETHERADDRL]; 1538 const uint8_t *dstaddr = ether_addr; 1539 mac_header_info_t non_ether_mhi; 1540 boolean_t is_unicast = B_FALSE; 1541 1542 mblk_t *mp = head; 1543 head = head->b_next; 1544 mp->b_next = NULL; 1545 const size_t sz1 = 1546 (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp); 1547 1548 if (is_ether) { 1549 uint32_t vlan_tci; 1550 1551 mac_ether_offload_info(mp, &meoi); 1552 if ((meoi.meoi_flags & MEOI_L2INFO_SET) == 0 || 1553 !mac_ether_l2_info(mp, ether_addr, &vlan_tci)) { 1554 mac_rx_drop_pkt(mac_srs, mp); 1555 continue; 1556 } 1557 1558 /* 1559 * Check if the VID of the packet, if any, belongs to 1560 * this client. Technically, if this packet came up via 1561 * a HW classified ring then we don't need to perform 1562 * this check. Perhaps a future optimization. 1563 */ 1564 if ((meoi.meoi_flags & MEOI_VLAN_TAGGED) != 0) { 1565 ASSERT3U(meoi.meoi_l2hlen, ==, 1566 sizeof (struct ether_vlan_header)); 1567 ASSERT3U(vlan_tci, <=, UINT16_MAX); 1568 1569 if (!mac_client_check_flow_vid(mcip, 1570 VLAN_ID(vlan_tci))) { 1571 mac_rx_drop_pkt(mac_srs, mp); 1572 continue; 1573 } 1574 } 1575 1576 is_unicast = ((ether_addr[0] & 0x01) == 0); 1577 } else { 1578 if (mac_header_info((mac_handle_t)mcip->mci_mip, 1579 mp, &non_ether_mhi) != 0) { 1580 mac_rx_drop_pkt(mac_srs, mp); 1581 continue; 1582 } 1583 1584 meoi.meoi_l2hlen = non_ether_mhi.mhi_hdrsize; 1585 meoi.meoi_l3proto = non_ether_mhi.mhi_bindsap; 1586 meoi.meoi_flags = MEOI_L2INFO_SET; 1587 (void) mac_partial_offload_info(mp, 0, &meoi); 1588 1589 is_unicast = 1590 (non_ether_mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST); 1591 dstaddr = non_ether_mhi.mhi_daddr; 1592 } 1593 1594 if (!dls_bypass) { 1595 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 1596 mac_ether_offload_info_t *, &meoi, 1597 mac_soft_ring_set_t *, mac_srs, pkt_type_t, OTH); 1598 FANOUT_ENQUEUE_MP(headmp[OTH], tailmp[OTH], 1599 cnt[OTH], bw_ctl, sz[OTH], sz1, mp); 1600 continue; 1601 } 1602 1603 ASSERT((meoi.meoi_flags & MEOI_L2INFO_SET) != 0); 1604 1605 boolean_t is_fastpath = B_FALSE; 1606 1607 if (meoi.meoi_l3proto == ETHERTYPE_IP || 1608 meoi.meoi_l3proto == ETHERTYPE_IPV6) { 1609 /* 1610 * If we are H/W classified, but we have promisc 1611 * on, then we need to check for the unicast address. 1612 */ 1613 if (hw_classified && mcip->mci_promisc_list != NULL) { 1614 mac_address_t *map; 1615 1616 rw_enter(&mcip->mci_rw_lock, RW_READER); 1617 map = mcip->mci_unicast; 1618 if (bcmp(dstaddr, map->ma_addr, 1619 map->ma_len) == 0) 1620 is_fastpath = B_TRUE; 1621 rw_exit(&mcip->mci_rw_lock); 1622 } else if (is_unicast) { 1623 is_fastpath = B_TRUE; 1624 } 1625 } 1626 1627 /* 1628 * This needs to become a contract with the driver for 1629 * the fast path. 1630 * 1631 * In the normal case the packet will have at least the L2 1632 * header and the IP + Transport header in the same mblk. 1633 * This is usually the case when the NIC driver sends up 1634 * the packet. This is also true when the stack generates 1635 * a packet that is looped back and when the stack uses the 1636 * fastpath mechanism. The normal case is optimized for 1637 * performance and may bypass DLS. All other cases go through 1638 * the 'OTH' type path without DLS bypass. 1639 */ 1640 if (is_fastpath) { 1641 if ((meoi.meoi_flags & MEOI_L3INFO_SET) == 0 || 1642 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0) { 1643 is_fastpath = B_FALSE; 1644 } 1645 if (DB_TYPE(mp) != M_DATA || DB_REF(mp) != 1) { 1646 is_fastpath = B_FALSE; 1647 } 1648 1649 const size_t total_hdr_len = meoi.meoi_l2hlen 1650 + meoi.meoi_l3hlen + meoi.meoi_l4hlen; 1651 1652 if (!OK_32PTR(mp->b_rptr + meoi.meoi_l2hlen) || 1653 total_hdr_len > MBLKL(mp)) { 1654 is_fastpath = B_FALSE; 1655 } 1656 } 1657 1658 if (!is_fastpath) { 1659 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 1660 mac_ether_offload_info_t *, &meoi, 1661 mac_soft_ring_set_t *, mac_srs, pkt_type_t, OTH); 1662 FANOUT_ENQUEUE_MP(headmp[OTH], tailmp[OTH], 1663 cnt[OTH], bw_ctl, sz[OTH], sz1, mp); 1664 continue; 1665 } 1666 1667 /* 1668 * Determine the type from the IP protocol value. If classified 1669 * as TCP or UDP, then update the read pointer to the beginning 1670 * of the IP header. Otherwise leave the message as is for 1671 * further processing by DLS. 1672 */ 1673 pkt_type_t type = OTH; 1674 switch (meoi.meoi_l4proto) { 1675 case IPPROTO_TCP: 1676 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ? 1677 V6_TCP : V4_TCP; 1678 mp->b_rptr += meoi.meoi_l2hlen; 1679 break; 1680 case IPPROTO_UDP: 1681 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ? 1682 V6_UDP : V4_UDP; 1683 mp->b_rptr += meoi.meoi_l2hlen; 1684 break; 1685 default: 1686 break; 1687 } 1688 1689 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 1690 mac_ether_offload_info_t *, &meoi, mac_soft_ring_set_t *, 1691 mac_srs, pkt_type_t, type); 1692 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], cnt[type], 1693 bw_ctl, sz[type], sz1, mp); 1694 } 1695 1696 for (pkt_type_t type = V4_TCP; type < UNDEF; type++) { 1697 if (headmp[type] != NULL) { 1698 mac_soft_ring_t *softring; 1699 1700 ASSERT(tailmp[type]->b_next == NULL); 1701 switch (type) { 1702 case V4_TCP: 1703 softring = mac_srs->srs_tcp_soft_rings[0]; 1704 break; 1705 case V6_TCP: 1706 softring = mac_srs->srs_tcp6_soft_rings[0]; 1707 break; 1708 case V4_UDP: 1709 softring = mac_srs->srs_udp_soft_rings[0]; 1710 break; 1711 case V6_UDP: 1712 softring = mac_srs->srs_udp6_soft_rings[0]; 1713 break; 1714 case OTH: 1715 softring = mac_srs->srs_oth_soft_rings[0]; 1716 } 1717 mac_rx_soft_ring_process(mcip, softring, 1718 headmp[type], tailmp[type], cnt[type], sz[type]); 1719 } 1720 } 1721 } 1722 1723 int fanout_unaligned = 0; 1724 1725 /* 1726 * The fanout routine for any clients with DLS bypass disabled or for 1727 * traffic classified as "other". Returns -1 on an error (drop the 1728 * packet due to a malformed packet), 0 on success, with values 1729 * written in *indx and *type. 1730 */ 1731 static int 1732 mac_rx_srs_long_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *mp, 1733 uint32_t sap, size_t hdrsize, pkt_type_t *type, uint_t *indx) 1734 { 1735 ip6_t *ip6h; 1736 ipha_t *ipha; 1737 uint8_t *whereptr; 1738 uint_t hash; 1739 uint16_t remlen; 1740 uint8_t nexthdr; 1741 uint16_t hdr_len; 1742 uint32_t src_val, dst_val; 1743 boolean_t modifiable = B_TRUE; 1744 boolean_t v6; 1745 1746 ASSERT(MBLKL(mp) >= hdrsize); 1747 1748 if (sap == ETHERTYPE_IPV6) { 1749 v6 = B_TRUE; 1750 hdr_len = IPV6_HDR_LEN; 1751 } else if (sap == ETHERTYPE_IP) { 1752 v6 = B_FALSE; 1753 hdr_len = IP_SIMPLE_HDR_LENGTH; 1754 } else { 1755 *indx = 0; 1756 *type = OTH; 1757 return (0); 1758 } 1759 1760 ip6h = (ip6_t *)(mp->b_rptr + hdrsize); 1761 ipha = (ipha_t *)ip6h; 1762 1763 if ((uint8_t *)ip6h == mp->b_wptr) { 1764 /* 1765 * The first mblk_t only includes the mac header. 1766 * Note that it is safe to change the mp pointer here, 1767 * as the subsequent operation does not assume mp 1768 * points to the start of the mac header. 1769 */ 1770 mp = mp->b_cont; 1771 1772 /* 1773 * Make sure the IP header points to an entire one. 1774 */ 1775 if (mp == NULL) 1776 return (-1); 1777 1778 if (MBLKL(mp) < hdr_len) { 1779 modifiable = (DB_REF(mp) == 1); 1780 1781 if (modifiable && !pullupmsg(mp, hdr_len)) 1782 return (-1); 1783 } 1784 1785 ip6h = (ip6_t *)mp->b_rptr; 1786 ipha = (ipha_t *)ip6h; 1787 } 1788 1789 if (!modifiable || !(OK_32PTR((char *)ip6h)) || 1790 ((uint8_t *)ip6h + hdr_len > mp->b_wptr)) { 1791 /* 1792 * If either the IP header is not aligned, or it does not hold 1793 * the complete simple structure (a pullupmsg() is not an 1794 * option since it would result in an unaligned IP header), 1795 * fanout to the default ring. 1796 * 1797 * Note that this may cause packet reordering. 1798 */ 1799 *indx = 0; 1800 *type = OTH; 1801 fanout_unaligned++; 1802 return (0); 1803 } 1804 1805 /* 1806 * Extract next-header, full header length, and source-hash value 1807 * using v4/v6 specific fields. 1808 */ 1809 if (v6) { 1810 remlen = ntohs(ip6h->ip6_plen); 1811 nexthdr = ip6h->ip6_nxt; 1812 src_val = V4_PART_OF_V6(ip6h->ip6_src); 1813 dst_val = V4_PART_OF_V6(ip6h->ip6_dst); 1814 /* 1815 * Do src based fanout if below tunable is set to B_TRUE or 1816 * when mac_ip_hdr_length_v6() fails because of malformed 1817 * packets or because mblks need to be concatenated using 1818 * pullupmsg(). 1819 * 1820 * Perform a version check to prevent parsing weirdness... 1821 */ 1822 if (IPH_HDR_VERSION(ip6h) != IPV6_VERSION || 1823 !mac_ip_hdr_length_v6(ip6h, mp->b_wptr, &hdr_len, &nexthdr, 1824 NULL)) { 1825 goto src_dst_based_fanout; 1826 } 1827 } else { 1828 hdr_len = IPH_HDR_LENGTH(ipha); 1829 remlen = ntohs(ipha->ipha_length) - hdr_len; 1830 nexthdr = ipha->ipha_protocol; 1831 src_val = (uint32_t)ipha->ipha_src; 1832 dst_val = (uint32_t)ipha->ipha_dst; 1833 /* 1834 * Catch IPv4 fragment case here. IPv6 has nexthdr == FRAG 1835 * for its equivalent case. 1836 */ 1837 if ((ntohs(ipha->ipha_fragment_offset_and_flags) & 1838 (IPH_MF | IPH_OFFSET)) != 0) { 1839 goto src_dst_based_fanout; 1840 } 1841 } 1842 if (remlen < MIN_EHDR_LEN) 1843 return (-1); 1844 whereptr = (uint8_t *)ip6h + hdr_len; 1845 1846 /* If the transport is one of below, we do port/SPI based fanout */ 1847 switch (nexthdr) { 1848 case IPPROTO_TCP: 1849 case IPPROTO_UDP: 1850 case IPPROTO_SCTP: 1851 case IPPROTO_ESP: 1852 /* 1853 * If the ports or SPI in the transport header is not part of 1854 * the mblk, do src_based_fanout, instead of calling 1855 * pullupmsg(). 1856 */ 1857 if (mp->b_cont == NULL || whereptr + PORTS_SIZE <= mp->b_wptr) 1858 break; /* out of switch... */ 1859 /* FALLTHRU */ 1860 default: 1861 goto src_dst_based_fanout; 1862 } 1863 1864 switch (nexthdr) { 1865 case IPPROTO_TCP: 1866 hash = HASH_ADDR(src_val, dst_val, *(uint32_t *)whereptr); 1867 *indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count); 1868 *type = OTH; 1869 break; 1870 case IPPROTO_UDP: 1871 case IPPROTO_SCTP: 1872 case IPPROTO_ESP: 1873 if (mac_fanout_type == MAC_FANOUT_DEFAULT) { 1874 hash = HASH_ADDR(src_val, dst_val, 1875 *(uint32_t *)whereptr); 1876 *indx = COMPUTE_INDEX(hash, 1877 mac_srs->srs_udp_ring_count); 1878 } else { 1879 *indx = mac_srs->srs_ind % mac_srs->srs_udp_ring_count; 1880 mac_srs->srs_ind++; 1881 } 1882 *type = OTH; 1883 break; 1884 } 1885 return (0); 1886 1887 src_dst_based_fanout: 1888 hash = HASH_ADDR(src_val, dst_val, (uint32_t)0); 1889 *indx = COMPUTE_INDEX(hash, mac_srs->srs_oth_ring_count); 1890 *type = OTH; 1891 return (0); 1892 } 1893 1894 /* 1895 * This routine delivers packets destined for an SRS into a soft ring member 1896 * of the set. 1897 * 1898 * Given a chain of packets we need to split it up into multiple sub 1899 * chains: TCP, UDP or OTH soft ring. Instead of entering the soft 1900 * ring one packet at a time, we want to enter it in the form of a 1901 * chain otherwise we get this start/stop behaviour where the worker 1902 * thread goes to sleep and then next packet comes in forcing it to 1903 * wake up. 1904 * 1905 * Note: 1906 * Since we know what is the maximum fanout possible, we create a 2D array 1907 * of 'softring types * MAX_SR_FANOUT' for the head, tail, cnt and sz 1908 * variables so that we can enter the softrings with chain. We need the 1909 * MAX_SR_FANOUT so we can allocate the arrays on the stack (a kmem_alloc 1910 * for each packet would be expensive). If we ever want to have the 1911 * ability to have unlimited fanout, we should probably declare a head, 1912 * tail, cnt, sz with each soft ring (a data struct which contains a softring 1913 * along with these members) and create an array of this uber struct so we 1914 * don't have to do kmem_alloc. 1915 */ 1916 1917 static void 1918 mac_rx_srs_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head) 1919 { 1920 mblk_t *headmp[MAX_SR_TYPES][MAX_SR_FANOUT]; 1921 mblk_t *tailmp[MAX_SR_TYPES][MAX_SR_FANOUT]; 1922 int cnt[MAX_SR_TYPES][MAX_SR_FANOUT]; 1923 size_t sz[MAX_SR_TYPES][MAX_SR_FANOUT]; 1924 mac_client_impl_t *mcip = mac_srs->srs_mcip; 1925 1926 const boolean_t is_ether = 1927 (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER); 1928 const boolean_t bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0); 1929 1930 /* 1931 * If we don't have a Rx ring, S/W classification would have done 1932 * its job and its a packet meant for us. If we were polling on 1933 * the default ring (i.e. there was a ring assigned to this SRS), 1934 * then we need to make sure that the mac address really belongs 1935 * to us. 1936 */ 1937 const boolean_t hw_classified = mac_srs->srs_ring != NULL && 1938 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER; 1939 1940 /* 1941 * Some clients, such as non Ethernet, need DLS processing in 1942 * the Rx path. Such clients clear the SRST_DLS_BYPASS flag. 1943 * DLS bypass may also be disabled via the 1944 * MCIS_RX_BYPASS_DISABLE flag, but this is only consumed by 1945 * sun4v vsw currently. 1946 */ 1947 const boolean_t dls_bypass = 1948 ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) && 1949 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0); 1950 1951 /* 1952 * Since the softrings are never destroyed and we always 1953 * create equal number of softrings for TCP, UDP and rest, 1954 * its OK to check one of them for count and use it without 1955 * any lock. In future, if soft rings get destroyed because 1956 * of reduction in fanout, we will need to ensure that happens 1957 * behind the SRS_PROC. 1958 */ 1959 const int fanout_cnt = mac_srs->srs_tcp_ring_count; 1960 1961 bzero(headmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *)); 1962 bzero(tailmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *)); 1963 bzero(cnt, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (int)); 1964 bzero(sz, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (size_t)); 1965 1966 /* 1967 * We got a chain from SRS that we need to send to the soft rings. 1968 * Since squeues for TCP & IPv4 SAP poll their soft rings (for 1969 * performance reasons), we need to separate out v4_tcp, v4_udp 1970 * and the rest goes in other. 1971 */ 1972 while (head != NULL) { 1973 mac_ether_offload_info_t meoi = { 0 }; 1974 uint8_t ether_addr[ETHERADDRL]; 1975 const uint8_t *dstaddr = ether_addr; 1976 mac_header_info_t non_ether_mhi; 1977 pkt_type_t type; 1978 uint_t indx; 1979 boolean_t is_unicast = B_FALSE; 1980 1981 mblk_t *mp = head; 1982 head = head->b_next; 1983 mp->b_next = NULL; 1984 const size_t sz1 = 1985 (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp); 1986 1987 if (is_ether) { 1988 uint32_t vlan_tci; 1989 1990 /* 1991 * At this point we can be sure the packet at least 1992 * has an ether header. 1993 */ 1994 mac_ether_offload_info(mp, &meoi); 1995 if ((meoi.meoi_flags & MEOI_L2INFO_SET) == 0 || 1996 !mac_ether_l2_info(mp, ether_addr, &vlan_tci)) { 1997 mac_rx_drop_pkt(mac_srs, mp); 1998 continue; 1999 } 2000 2001 /* 2002 * Check if the VID of the packet, if any, belongs to 2003 * this client. Technically, if this packet came up via 2004 * a HW classified ring then we don't need to perform 2005 * this check. Perhaps a future optimization. 2006 */ 2007 if ((meoi.meoi_flags & MEOI_VLAN_TAGGED) != 0) { 2008 ASSERT3U(meoi.meoi_l2hlen, ==, 2009 sizeof (struct ether_vlan_header)); 2010 ASSERT3U(vlan_tci, <=, UINT16_MAX); 2011 2012 if (!mac_client_check_flow_vid(mcip, 2013 VLAN_ID(vlan_tci))) { 2014 mac_rx_drop_pkt(mac_srs, mp); 2015 continue; 2016 } 2017 } 2018 2019 is_unicast = (ether_addr[0] & 0x01) == 0; 2020 } else { 2021 if (mac_header_info((mac_handle_t)mcip->mci_mip, 2022 mp, &non_ether_mhi) != 0) { 2023 mac_rx_drop_pkt(mac_srs, mp); 2024 continue; 2025 } 2026 2027 meoi.meoi_l2hlen = non_ether_mhi.mhi_hdrsize; 2028 meoi.meoi_l3proto = non_ether_mhi.mhi_bindsap; 2029 meoi.meoi_flags = MEOI_L2INFO_SET; 2030 (void) mac_partial_offload_info(mp, 0, &meoi); 2031 2032 is_unicast = 2033 (non_ether_mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST); 2034 dstaddr = non_ether_mhi.mhi_daddr; 2035 } 2036 2037 if (!dls_bypass) { 2038 if (mac_rx_srs_long_fanout(mac_srs, mp, 2039 meoi.meoi_l3proto, meoi.meoi_l2hlen, 2040 &type, &indx) == -1) { 2041 mac_rx_drop_pkt(mac_srs, mp); 2042 continue; 2043 } 2044 2045 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 2046 mac_ether_offload_info_t *, &meoi, 2047 mac_soft_ring_set_t *, mac_srs, pkt_type_t, type); 2048 FANOUT_ENQUEUE_MP(headmp[type][indx], 2049 tailmp[type][indx], 2050 cnt[type][indx], bw_ctl, 2051 sz[type][indx], sz1, mp); 2052 continue; 2053 } 2054 2055 /* 2056 * While MEOI is unable to parse ESP headers, for the purposes 2057 * of classification here, we treat such packets like UDP, so we 2058 * can grant it a reprieve here. This is acceptable since we 2059 * will not go rooting around in the ESP headers. 2060 */ 2061 if ((meoi.meoi_flags & MEOI_L3INFO_SET) != 0 && 2062 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0 && 2063 meoi.meoi_l4proto == IPPROTO_ESP) { 2064 /* ESP header should consist of at least 8 octets */ 2065 meoi.meoi_l4hlen = 8; 2066 meoi.meoi_flags |= MEOI_L4INFO_SET; 2067 } 2068 2069 /* 2070 * If we are using the default Rx ring where H/W or S/W 2071 * classification has not happened, we need to verify if 2072 * this unicast packet really belongs to us. 2073 */ 2074 boolean_t is_fastpath = B_FALSE; 2075 if (meoi.meoi_l3proto == ETHERTYPE_IP || 2076 meoi.meoi_l3proto == ETHERTYPE_IPV6) { 2077 /* 2078 * If we are H/W classified, but we have promisc 2079 * on, then we need to check for the unicast address. 2080 */ 2081 if (hw_classified && mcip->mci_promisc_list != NULL) { 2082 mac_address_t *map; 2083 2084 rw_enter(&mcip->mci_rw_lock, RW_READER); 2085 map = mcip->mci_unicast; 2086 if (bcmp(dstaddr, map->ma_addr, 2087 map->ma_len) == 0) 2088 is_fastpath = B_TRUE; 2089 rw_exit(&mcip->mci_rw_lock); 2090 } else if (is_unicast) { 2091 is_fastpath = B_TRUE; 2092 } 2093 } 2094 2095 /* 2096 * Verify that the requirements for taking the fast path are all 2097 * still met. This needs to become a contract with the driver. 2098 */ 2099 if (is_fastpath) { 2100 if ((meoi.meoi_flags & MEOI_L3INFO_SET) == 0 || 2101 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0) { 2102 is_fastpath = B_FALSE; 2103 } 2104 if (DB_TYPE(mp) != M_DATA || DB_REF(mp) != 1) { 2105 is_fastpath = B_FALSE; 2106 } 2107 2108 const size_t total_hdr_len = meoi.meoi_l2hlen 2109 + meoi.meoi_l3hlen + meoi.meoi_l4hlen; 2110 2111 if (!OK_32PTR(mp->b_rptr + meoi.meoi_l2hlen) || 2112 total_hdr_len > MBLKL(mp)) { 2113 is_fastpath = B_FALSE; 2114 } 2115 2116 if ((meoi.meoi_flags & 2117 (MEOI_L3_FRAG_MORE | MEOI_L3_FRAG_OFFSET)) != 0) { 2118 is_fastpath = B_FALSE; 2119 } 2120 } 2121 switch (meoi.meoi_l4proto) { 2122 case IPPROTO_TCP: 2123 case IPPROTO_UDP: 2124 case IPPROTO_SCTP: 2125 case IPPROTO_ESP: 2126 if (is_fastpath) { 2127 /* 2128 * Since the above checks ensure that the first 2129 * mblk covers the L2-L4 headers, we can be 2130 * confident that the "ports" portion of the 2131 * hashing payload is covered too. 2132 */ 2133 ASSERT3U(meoi.meoi_l4hlen, >=, PORTS_SIZE); 2134 } 2135 break; 2136 default: 2137 break; 2138 } 2139 2140 if (!is_fastpath) { 2141 if (mac_rx_srs_long_fanout(mac_srs, mp, 2142 meoi.meoi_l3proto, meoi.meoi_l2hlen, 2143 &type, &indx) == -1) { 2144 mac_rx_drop_pkt(mac_srs, mp); 2145 continue; 2146 } 2147 2148 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 2149 mac_ether_offload_info_t *, &meoi, 2150 mac_soft_ring_set_t *, mac_srs, pkt_type_t, type); 2151 FANOUT_ENQUEUE_MP(headmp[type][indx], 2152 tailmp[type][indx], cnt[type][indx], bw_ctl, 2153 sz[type][indx], sz1, mp); 2154 continue; 2155 } 2156 2157 /* 2158 * By now, the fastpath requirements ensure that direct access 2159 * to the L3/L4 headers will fall safely within the mblk. 2160 */ 2161 const ipha_t *ipha = (ipha_t *)(mp->b_rptr + meoi.meoi_l2hlen); 2162 const ip6_t *ip6 = (ip6_t *)(mp->b_rptr + meoi.meoi_l2hlen); 2163 const uint32_t *ports = (uint32_t *) 2164 (mp->b_rptr + meoi.meoi_l2hlen + meoi.meoi_l3hlen); 2165 2166 /* 2167 * XXX-Sunay: We should hold srs_lock since ring_count 2168 * below can change. But if we are always called from 2169 * mac_rx_srs_drain and SRS_PROC is set, then we can 2170 * enforce that ring_count can't be changed i.e. 2171 * to change fanout type or ring count, the calling 2172 * thread needs to be behind SRS_PROC. 2173 */ 2174 uint_t hash; 2175 switch (meoi.meoi_l4proto) { 2176 case IPPROTO_TCP: 2177 /* 2178 * Note that for ESP, we fanout on SPI and it is at the 2179 * same offset as the 2x16-bit ports. So it is clumped 2180 * along with TCP, UDP and SCTP. 2181 */ 2182 if (meoi.meoi_l3proto == ETHERTYPE_IP) { 2183 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst, 2184 *ports); 2185 type = V4_TCP; 2186 } 2187 if (meoi.meoi_l3proto == ETHERTYPE_IPV6) { 2188 hash = HASH_ADDR6(ip6->ip6_src, ip6->ip6_dst, 2189 *ports); 2190 type = V6_TCP; 2191 } 2192 indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count); 2193 mp->b_rptr += meoi.meoi_l2hlen; 2194 break; 2195 case IPPROTO_UDP: 2196 case IPPROTO_SCTP: 2197 case IPPROTO_ESP: 2198 if (mac_fanout_type == MAC_FANOUT_DEFAULT) { 2199 if (meoi.meoi_l3proto == ETHERTYPE_IP) { 2200 hash = HASH_ADDR(ipha->ipha_src, 2201 ipha->ipha_dst, *ports); 2202 } 2203 if (meoi.meoi_l3proto == ETHERTYPE_IPV6) { 2204 hash = HASH_ADDR6(ip6->ip6_src, 2205 ip6->ip6_dst, *ports); 2206 } 2207 indx = COMPUTE_INDEX(hash, 2208 mac_srs->srs_udp_ring_count); 2209 } else { 2210 indx = mac_srs->srs_ind % 2211 mac_srs->srs_udp_ring_count; 2212 mac_srs->srs_ind++; 2213 } 2214 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ? 2215 V6_UDP : V4_UDP; 2216 mp->b_rptr += meoi.meoi_l2hlen; 2217 break; 2218 default: 2219 indx = 0; 2220 type = OTH; 2221 } 2222 2223 DTRACE_PROBE4(rx__fanout, mblk_t *, mp, 2224 mac_ether_offload_info_t *, &meoi, mac_soft_ring_set_t *, 2225 mac_srs, pkt_type_t, type); 2226 FANOUT_ENQUEUE_MP(headmp[type][indx], tailmp[type][indx], 2227 cnt[type][indx], bw_ctl, sz[type][indx], sz1, mp); 2228 } 2229 2230 for (pkt_type_t type = V4_TCP; type < UNDEF; type++) { 2231 for (int i = 0; i < fanout_cnt; i++) { 2232 if (headmp[type][i] != NULL) { 2233 mac_soft_ring_t *softring; 2234 2235 ASSERT(tailmp[type][i]->b_next == NULL); 2236 switch (type) { 2237 case V4_TCP: 2238 softring = 2239 mac_srs->srs_tcp_soft_rings[i]; 2240 break; 2241 case V6_TCP: 2242 softring = 2243 mac_srs->srs_tcp6_soft_rings[i]; 2244 break; 2245 case V4_UDP: 2246 softring = 2247 mac_srs->srs_udp_soft_rings[i]; 2248 break; 2249 case V6_UDP: 2250 softring = 2251 mac_srs->srs_udp6_soft_rings[i]; 2252 break; 2253 case OTH: 2254 softring = 2255 mac_srs->srs_oth_soft_rings[i]; 2256 break; 2257 } 2258 mac_rx_soft_ring_process(mcip, 2259 softring, headmp[type][i], tailmp[type][i], 2260 cnt[type][i], sz[type][i]); 2261 } 2262 } 2263 } 2264 } 2265 2266 #define SRS_BYTES_TO_PICKUP 150000 2267 ssize_t max_bytes_to_pickup = SRS_BYTES_TO_PICKUP; 2268 2269 /* 2270 * mac_rx_srs_poll_ring 2271 * 2272 * This SRS Poll thread uses this routine to poll the underlying hardware 2273 * Rx ring to get a chain of packets. It can inline process that chain 2274 * if mac_latency_optimize is set (default) or signal the SRS worker thread 2275 * to do the remaining processing. 2276 * 2277 * Since packets come in the system via interrupt or poll path, we also 2278 * update the stats and deal with promiscous clients here. 2279 */ 2280 void 2281 mac_rx_srs_poll_ring(mac_soft_ring_set_t *mac_srs) 2282 { 2283 kmutex_t *lock = &mac_srs->srs_lock; 2284 kcondvar_t *async = &mac_srs->srs_cv; 2285 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2286 mblk_t *head, *tail, *mp; 2287 callb_cpr_t cprinfo; 2288 ssize_t bytes_to_pickup; 2289 size_t sz; 2290 int count; 2291 mac_client_impl_t *smcip; 2292 2293 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "mac_srs_poll"); 2294 mutex_enter(lock); 2295 2296 start: 2297 for (;;) { 2298 if (mac_srs->srs_state & SRS_PAUSE) 2299 goto done; 2300 2301 CALLB_CPR_SAFE_BEGIN(&cprinfo); 2302 cv_wait(async, lock); 2303 CALLB_CPR_SAFE_END(&cprinfo, lock); 2304 2305 if (mac_srs->srs_state & SRS_PAUSE) 2306 goto done; 2307 2308 check_again: 2309 if (mac_srs->srs_type & SRST_BW_CONTROL) { 2310 /* 2311 * We pick as many bytes as we are allowed to queue. 2312 * Its possible that we will exceed the total 2313 * packets queued in case this SRS is part of the 2314 * Rx ring group since > 1 poll thread can be pulling 2315 * upto the max allowed packets at the same time 2316 * but that should be OK. 2317 */ 2318 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2319 bytes_to_pickup = 2320 mac_srs->srs_bw->mac_bw_drop_threshold - 2321 mac_srs->srs_bw->mac_bw_sz; 2322 /* 2323 * We shouldn't have been signalled if we 2324 * have 0 or less bytes to pick but since 2325 * some of the bytes accounting is driver 2326 * dependant, we do the safety check. 2327 */ 2328 if (bytes_to_pickup < 0) 2329 bytes_to_pickup = 0; 2330 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2331 } else { 2332 /* 2333 * ToDO: Need to change the polling API 2334 * to add a packet count and a flag which 2335 * tells the driver whether we want packets 2336 * based on a count, or bytes, or all the 2337 * packets queued in the driver/HW. This 2338 * way, we never have to check the limits 2339 * on poll path. We truly let only as many 2340 * packets enter the system as we are willing 2341 * to process or queue. 2342 * 2343 * Something along the lines of 2344 * pkts_to_pickup = mac_soft_ring_max_q_cnt - 2345 * mac_srs->srs_poll_pkt_cnt 2346 */ 2347 2348 /* 2349 * Since we are not doing B/W control, pick 2350 * as many packets as allowed. 2351 */ 2352 bytes_to_pickup = max_bytes_to_pickup; 2353 } 2354 2355 /* Poll the underlying Hardware */ 2356 mutex_exit(lock); 2357 head = MAC_HWRING_POLL(mac_srs->srs_ring, (int)bytes_to_pickup); 2358 mutex_enter(lock); 2359 2360 ASSERT((mac_srs->srs_state & SRS_POLL_THR_OWNER) == 2361 SRS_POLL_THR_OWNER); 2362 2363 mp = tail = head; 2364 count = 0; 2365 sz = 0; 2366 while (mp != NULL) { 2367 tail = mp; 2368 sz += msgdsize(mp); 2369 mp = mp->b_next; 2370 count++; 2371 } 2372 2373 if (head != NULL) { 2374 tail->b_next = NULL; 2375 smcip = mac_srs->srs_mcip; 2376 2377 SRS_RX_STAT_UPDATE(mac_srs, pollbytes, sz); 2378 SRS_RX_STAT_UPDATE(mac_srs, pollcnt, count); 2379 2380 /* 2381 * If there are any promiscuous mode callbacks 2382 * defined for this MAC client, pass them a copy 2383 * if appropriate and also update the counters. 2384 */ 2385 if (smcip != NULL) { 2386 if (smcip->mci_mip->mi_promisc_list != NULL) { 2387 mutex_exit(lock); 2388 mac_promisc_dispatch(smcip->mci_mip, 2389 head, NULL, B_FALSE); 2390 mutex_enter(lock); 2391 } 2392 } 2393 if (mac_srs->srs_type & SRST_BW_CONTROL) { 2394 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2395 mac_srs->srs_bw->mac_bw_polled += sz; 2396 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2397 } 2398 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, 2399 count, sz); 2400 if (count <= 10) 2401 srs_rx->sr_stat.mrs_chaincntundr10++; 2402 else if (count > 10 && count <= 50) 2403 srs_rx->sr_stat.mrs_chaincnt10to50++; 2404 else 2405 srs_rx->sr_stat.mrs_chaincntover50++; 2406 } 2407 2408 /* 2409 * We are guaranteed that SRS_PROC will be set if we 2410 * are here. Also, poll thread gets to run only if 2411 * the drain was being done by a worker thread although 2412 * its possible that worker thread is still running 2413 * and poll thread was sent down to keep the pipeline 2414 * going instead of doing a complete drain and then 2415 * trying to poll the NIC. 2416 * 2417 * So we need to check SRS_WORKER flag to make sure 2418 * that the worker thread is not processing the queue 2419 * in parallel to us. The flags and conditions are 2420 * protected by the srs_lock to prevent any race. We 2421 * ensure that we don't drop the srs_lock from now 2422 * till the end and similarly we don't drop the srs_lock 2423 * in mac_rx_srs_drain() till similar condition check 2424 * are complete. The mac_rx_srs_drain() needs to ensure 2425 * that SRS_WORKER flag remains set as long as its 2426 * processing the queue. 2427 */ 2428 if (!(mac_srs->srs_state & SRS_WORKER) && 2429 (mac_srs->srs_first != NULL)) { 2430 /* 2431 * We have packets to process and worker thread 2432 * is not running. Check to see if poll thread is 2433 * allowed to process. 2434 */ 2435 if (mac_srs->srs_state & SRS_LATENCY_OPT) { 2436 mac_srs->srs_drain_func(mac_srs, SRS_POLL_PROC); 2437 if (!(mac_srs->srs_state & SRS_PAUSE) && 2438 srs_rx->sr_poll_pkt_cnt <= 2439 srs_rx->sr_lowat) { 2440 srs_rx->sr_poll_again++; 2441 goto check_again; 2442 } 2443 /* 2444 * We are already above low water mark 2445 * so stay in the polling mode but no 2446 * need to poll. Once we dip below 2447 * the polling threshold, the processing 2448 * thread (soft ring) will signal us 2449 * to poll again (MAC_UPDATE_SRS_COUNT) 2450 */ 2451 srs_rx->sr_poll_drain_no_poll++; 2452 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS); 2453 /* 2454 * In B/W control case, its possible 2455 * that the backlog built up due to 2456 * B/W limit being reached and packets 2457 * are queued only in SRS. In this case, 2458 * we should schedule worker thread 2459 * since no one else will wake us up. 2460 */ 2461 if ((mac_srs->srs_type & SRST_BW_CONTROL) && 2462 (mac_srs->srs_tid == NULL)) { 2463 mac_srs->srs_tid = 2464 timeout(mac_srs_fire, mac_srs, 1); 2465 srs_rx->sr_poll_worker_wakeup++; 2466 } 2467 } else { 2468 /* 2469 * Wakeup the worker thread for more processing. 2470 * We optimize for throughput in this case. 2471 */ 2472 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS); 2473 MAC_SRS_WORKER_WAKEUP(mac_srs); 2474 srs_rx->sr_poll_sig_worker++; 2475 } 2476 } else if ((mac_srs->srs_first == NULL) && 2477 !(mac_srs->srs_state & SRS_WORKER)) { 2478 /* 2479 * There is nothing queued in SRS and 2480 * no worker thread running. Plus we 2481 * didn't get anything from the H/W 2482 * as well (head == NULL); 2483 */ 2484 ASSERT(head == NULL); 2485 mac_srs->srs_state &= 2486 ~(SRS_PROC|SRS_GET_PKTS); 2487 2488 /* 2489 * If we have a packets in soft ring, don't allow 2490 * more packets to come into this SRS by keeping the 2491 * interrupts off but not polling the H/W. The 2492 * poll thread will get signaled as soon as 2493 * srs_poll_pkt_cnt dips below poll threshold. 2494 */ 2495 if (srs_rx->sr_poll_pkt_cnt == 0) { 2496 srs_rx->sr_poll_intr_enable++; 2497 MAC_SRS_POLLING_OFF(mac_srs); 2498 } else { 2499 /* 2500 * We know nothing is queued in SRS 2501 * since we are here after checking 2502 * srs_first is NULL. The backlog 2503 * is entirely due to packets queued 2504 * in Soft ring which will wake us up 2505 * and get the interface out of polling 2506 * mode once the backlog dips below 2507 * sr_poll_thres. 2508 */ 2509 srs_rx->sr_poll_no_poll++; 2510 } 2511 } else { 2512 /* 2513 * Worker thread is already running. 2514 * Nothing much to do. If the polling 2515 * was enabled, worker thread will deal 2516 * with that. 2517 */ 2518 mac_srs->srs_state &= ~SRS_GET_PKTS; 2519 srs_rx->sr_poll_goto_sleep++; 2520 } 2521 } 2522 done: 2523 mac_srs->srs_state |= SRS_POLL_THR_QUIESCED; 2524 cv_signal(&mac_srs->srs_async); 2525 /* 2526 * If this is a temporary quiesce then wait for the restart signal 2527 * from the srs worker. Then clear the flags and signal the srs worker 2528 * to ensure a positive handshake and go back to start. 2529 */ 2530 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_POLL_THR_RESTART))) 2531 cv_wait(async, lock); 2532 if (mac_srs->srs_state & SRS_POLL_THR_RESTART) { 2533 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED)); 2534 mac_srs->srs_state &= 2535 ~(SRS_POLL_THR_QUIESCED | SRS_POLL_THR_RESTART); 2536 cv_signal(&mac_srs->srs_async); 2537 goto start; 2538 } else { 2539 mac_srs->srs_state |= SRS_POLL_THR_EXITED; 2540 cv_signal(&mac_srs->srs_async); 2541 CALLB_CPR_EXIT(&cprinfo); 2542 thread_exit(); 2543 } 2544 } 2545 2546 /* 2547 * mac_srs_pick_chain 2548 * 2549 * In Bandwidth control case, checks how many packets can be processed 2550 * and return them in a sub chain. 2551 */ 2552 static mblk_t * 2553 mac_srs_pick_chain(mac_soft_ring_set_t *mac_srs, mblk_t **chain_tail, 2554 size_t *chain_sz, int *chain_cnt) 2555 { 2556 mblk_t *head = NULL; 2557 mblk_t *tail = NULL; 2558 size_t sz; 2559 size_t tsz = 0; 2560 int cnt = 0; 2561 mblk_t *mp; 2562 2563 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2564 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2565 if (((mac_srs->srs_bw->mac_bw_used + mac_srs->srs_size) <= 2566 mac_srs->srs_bw->mac_bw_limit) || 2567 (mac_srs->srs_bw->mac_bw_limit == 0)) { 2568 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2569 head = mac_srs->srs_first; 2570 mac_srs->srs_first = NULL; 2571 *chain_tail = mac_srs->srs_last; 2572 mac_srs->srs_last = NULL; 2573 *chain_sz = mac_srs->srs_size; 2574 *chain_cnt = mac_srs->srs_count; 2575 mac_srs->srs_count = 0; 2576 mac_srs->srs_size = 0; 2577 return (head); 2578 } 2579 2580 /* 2581 * Can't clear the entire backlog. 2582 * Need to find how many packets to pick 2583 */ 2584 ASSERT(MUTEX_HELD(&mac_srs->srs_bw->mac_bw_lock)); 2585 while ((mp = mac_srs->srs_first) != NULL) { 2586 sz = msgdsize(mp); 2587 if ((tsz + sz + mac_srs->srs_bw->mac_bw_used) > 2588 mac_srs->srs_bw->mac_bw_limit) { 2589 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) 2590 mac_srs->srs_bw->mac_bw_state |= 2591 SRS_BW_ENFORCED; 2592 break; 2593 } 2594 2595 /* 2596 * The _size & cnt is decremented from the softrings 2597 * when they send up the packet for polling to work 2598 * properly. 2599 */ 2600 tsz += sz; 2601 cnt++; 2602 mac_srs->srs_count--; 2603 mac_srs->srs_size -= sz; 2604 if (tail != NULL) 2605 tail->b_next = mp; 2606 else 2607 head = mp; 2608 tail = mp; 2609 mac_srs->srs_first = mac_srs->srs_first->b_next; 2610 } 2611 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2612 if (mac_srs->srs_first == NULL) 2613 mac_srs->srs_last = NULL; 2614 2615 if (tail != NULL) 2616 tail->b_next = NULL; 2617 *chain_tail = tail; 2618 *chain_cnt = cnt; 2619 *chain_sz = tsz; 2620 2621 return (head); 2622 } 2623 2624 /* 2625 * mac_rx_srs_drain 2626 * 2627 * The SRS drain routine. Gets to run to clear the queue. Any thread 2628 * (worker, interrupt, poll) can call this based on processing model. 2629 * The first thing we do is disable interrupts if possible and then 2630 * drain the queue. we also try to poll the underlying hardware if 2631 * there is a dedicated hardware Rx ring assigned to this SRS. 2632 * 2633 * There is a equivalent drain routine in bandwidth control mode 2634 * mac_rx_srs_drain_bw. There is some code duplication between the two 2635 * routines but they are highly performance sensitive and are easier 2636 * to read/debug if they stay separate. Any code changes here might 2637 * also apply to mac_rx_srs_drain_bw as well. 2638 */ 2639 void 2640 mac_rx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 2641 { 2642 mblk_t *head; 2643 mblk_t *tail; 2644 timeout_id_t tid; 2645 int cnt = 0; 2646 mac_client_impl_t *mcip = mac_srs->srs_mcip; 2647 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2648 2649 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2650 ASSERT(!(mac_srs->srs_type & SRST_BW_CONTROL)); 2651 2652 /* If we are blanked i.e. can't do upcalls, then we are done */ 2653 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) { 2654 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) || 2655 (mac_srs->srs_state & SRS_PAUSE)); 2656 goto out; 2657 } 2658 2659 if (mac_srs->srs_first == NULL) 2660 goto out; 2661 2662 if (!(mac_srs->srs_state & SRS_LATENCY_OPT) && 2663 (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)) { 2664 /* 2665 * In the normal case, the SRS worker thread does no 2666 * work and we wait for a backlog to build up before 2667 * we switch into polling mode. In case we are 2668 * optimizing for throughput, we use the worker thread 2669 * as well. The goal is to let worker thread process 2670 * the queue and poll thread to feed packets into 2671 * the queue. As such, we should signal the poll 2672 * thread to try and get more packets. 2673 * 2674 * We could have pulled this check in the POLL_RING 2675 * macro itself but keeping it explicit here makes 2676 * the architecture more human understandable. 2677 */ 2678 MAC_SRS_POLL_RING(mac_srs); 2679 } 2680 2681 again: 2682 head = mac_srs->srs_first; 2683 mac_srs->srs_first = NULL; 2684 tail = mac_srs->srs_last; 2685 mac_srs->srs_last = NULL; 2686 cnt = mac_srs->srs_count; 2687 mac_srs->srs_count = 0; 2688 2689 ASSERT(head != NULL); 2690 ASSERT(tail != NULL); 2691 2692 if ((tid = mac_srs->srs_tid) != NULL) 2693 mac_srs->srs_tid = NULL; 2694 2695 mac_srs->srs_state |= (SRS_PROC|proc_type); 2696 2697 /* 2698 * mcip is NULL for broadcast and multicast flows. The promisc 2699 * callbacks for broadcast and multicast packets are delivered from 2700 * mac_rx() and we don't need to worry about that case in this path 2701 */ 2702 if (mcip != NULL) { 2703 if (mcip->mci_promisc_list != NULL) { 2704 mutex_exit(&mac_srs->srs_lock); 2705 mac_promisc_client_dispatch(mcip, head); 2706 mutex_enter(&mac_srs->srs_lock); 2707 } 2708 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) { 2709 mutex_exit(&mac_srs->srs_lock); 2710 mac_protect_intercept_dynamic(mcip, head); 2711 mutex_enter(&mac_srs->srs_lock); 2712 } 2713 } 2714 2715 /* 2716 * Check if SRS itself is doing the processing. This direct 2717 * path applies only when subflows are present. 2718 */ 2719 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) { 2720 mac_direct_rx_t proc; 2721 void *arg1; 2722 mac_resource_handle_t arg2; 2723 2724 /* 2725 * This is the case when a Rx is directly 2726 * assigned and we have a fully classified 2727 * protocol chain. We can deal with it in 2728 * one shot. 2729 */ 2730 proc = srs_rx->sr_func; 2731 arg1 = srs_rx->sr_arg1; 2732 arg2 = srs_rx->sr_arg2; 2733 2734 mac_srs->srs_state |= SRS_CLIENT_PROC; 2735 mutex_exit(&mac_srs->srs_lock); 2736 if (tid != NULL) { 2737 (void) untimeout(tid); 2738 tid = NULL; 2739 } 2740 2741 proc(arg1, arg2, head, NULL); 2742 /* 2743 * Decrement the size and count here itelf 2744 * since the packet has been processed. 2745 */ 2746 mutex_enter(&mac_srs->srs_lock); 2747 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 2748 if (mac_srs->srs_state & SRS_CLIENT_WAIT) 2749 cv_signal(&mac_srs->srs_client_cv); 2750 mac_srs->srs_state &= ~SRS_CLIENT_PROC; 2751 } else { 2752 /* Some kind of softrings based fanout is required */ 2753 mutex_exit(&mac_srs->srs_lock); 2754 if (tid != NULL) { 2755 (void) untimeout(tid); 2756 tid = NULL; 2757 } 2758 2759 /* 2760 * Since the fanout routines can deal with chains, 2761 * shoot the entire chain up. 2762 */ 2763 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP) 2764 mac_rx_srs_fanout(mac_srs, head); 2765 else 2766 mac_rx_srs_proto_fanout(mac_srs, head); 2767 mutex_enter(&mac_srs->srs_lock); 2768 } 2769 2770 if (!(mac_srs->srs_state & (SRS_BLANK|SRS_PAUSE)) && 2771 (mac_srs->srs_first != NULL)) { 2772 /* 2773 * More packets arrived while we were clearing the 2774 * SRS. This can be possible because of one of 2775 * three conditions below: 2776 * 1) The driver is using multiple worker threads 2777 * to send the packets to us. 2778 * 2) The driver has a race in switching 2779 * between interrupt and polling mode or 2780 * 3) Packets are arriving in this SRS via the 2781 * S/W classification as well. 2782 * 2783 * We should switch to polling mode and see if we 2784 * need to send the poll thread down. Also, signal 2785 * the worker thread to process whats just arrived. 2786 */ 2787 MAC_SRS_POLLING_ON(mac_srs); 2788 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) { 2789 srs_rx->sr_drain_poll_sig++; 2790 MAC_SRS_POLL_RING(mac_srs); 2791 } 2792 2793 /* 2794 * If we didn't signal the poll thread, we need 2795 * to deal with the pending packets ourselves. 2796 */ 2797 if (proc_type == SRS_WORKER) { 2798 srs_rx->sr_drain_again++; 2799 goto again; 2800 } else { 2801 srs_rx->sr_drain_worker_sig++; 2802 cv_signal(&mac_srs->srs_async); 2803 } 2804 } 2805 2806 out: 2807 if (mac_srs->srs_state & SRS_GET_PKTS) { 2808 /* 2809 * Poll thread is already running. Leave the 2810 * SRS_RPOC set and hand over the control to 2811 * poll thread. 2812 */ 2813 mac_srs->srs_state &= ~proc_type; 2814 srs_rx->sr_drain_poll_running++; 2815 return; 2816 } 2817 2818 /* 2819 * Even if there are no packets queued in SRS, we 2820 * need to make sure that the shared counter is 2821 * clear and any associated softrings have cleared 2822 * all the backlog. Otherwise, leave the interface 2823 * in polling mode and the poll thread will get 2824 * signalled once the count goes down to zero. 2825 * 2826 * If someone is already draining the queue (SRS_PROC is 2827 * set) when the srs_poll_pkt_cnt goes down to zero, 2828 * then it means that drain is already running and we 2829 * will turn off polling at that time if there is 2830 * no backlog. 2831 * 2832 * As long as there are packets queued either 2833 * in soft ring set or its soft rings, we will leave 2834 * the interface in polling mode (even if the drain 2835 * was done being the interrupt thread). We signal 2836 * the poll thread as well if we have dipped below 2837 * low water mark. 2838 * 2839 * NOTE: We can't use the MAC_SRS_POLLING_ON macro 2840 * since that turn polling on only for worker thread. 2841 * Its not worth turning polling on for interrupt 2842 * thread (since NIC will not issue another interrupt) 2843 * unless a backlog builds up. 2844 */ 2845 if ((srs_rx->sr_poll_pkt_cnt > 0) && 2846 (mac_srs->srs_state & SRS_POLLING_CAPAB)) { 2847 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 2848 srs_rx->sr_drain_keep_polling++; 2849 MAC_SRS_POLLING_ON(mac_srs); 2850 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) 2851 MAC_SRS_POLL_RING(mac_srs); 2852 return; 2853 } 2854 2855 /* Nothing else to do. Get out of poll mode */ 2856 MAC_SRS_POLLING_OFF(mac_srs); 2857 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 2858 srs_rx->sr_drain_finish_intr++; 2859 } 2860 2861 /* 2862 * mac_rx_srs_drain_bw 2863 * 2864 * The SRS BW drain routine. Gets to run to clear the queue. Any thread 2865 * (worker, interrupt, poll) can call this based on processing model. 2866 * The first thing we do is disable interrupts if possible and then 2867 * drain the queue. we also try to poll the underlying hardware if 2868 * there is a dedicated hardware Rx ring assigned to this SRS. 2869 * 2870 * There is a equivalent drain routine in non bandwidth control mode 2871 * mac_rx_srs_drain. There is some code duplication between the two 2872 * routines but they are highly performance sensitive and are easier 2873 * to read/debug if they stay separate. Any code changes here might 2874 * also apply to mac_rx_srs_drain as well. 2875 */ 2876 void 2877 mac_rx_srs_drain_bw(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 2878 { 2879 mblk_t *head; 2880 mblk_t *tail; 2881 timeout_id_t tid; 2882 size_t sz = 0; 2883 int cnt = 0; 2884 mac_client_impl_t *mcip = mac_srs->srs_mcip; 2885 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2886 clock_t now; 2887 2888 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2889 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL); 2890 again: 2891 /* Check if we are doing B/W control */ 2892 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2893 now = ddi_get_lbolt(); 2894 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 2895 mac_srs->srs_bw->mac_bw_curr_time = now; 2896 mac_srs->srs_bw->mac_bw_used = 0; 2897 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) 2898 mac_srs->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; 2899 } else if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) { 2900 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2901 goto done; 2902 } else if (mac_srs->srs_bw->mac_bw_used > 2903 mac_srs->srs_bw->mac_bw_limit) { 2904 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 2905 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2906 goto done; 2907 } 2908 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2909 2910 /* If we are blanked i.e. can't do upcalls, then we are done */ 2911 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) { 2912 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) || 2913 (mac_srs->srs_state & SRS_PAUSE)); 2914 goto done; 2915 } 2916 2917 sz = 0; 2918 cnt = 0; 2919 if ((head = mac_srs_pick_chain(mac_srs, &tail, &sz, &cnt)) == NULL) { 2920 /* 2921 * We couldn't pick up a single packet. 2922 */ 2923 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2924 if ((mac_srs->srs_bw->mac_bw_used == 0) && 2925 (mac_srs->srs_size != 0) && 2926 !(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 2927 /* 2928 * Seems like configured B/W doesn't 2929 * even allow processing of 1 packet 2930 * per tick. 2931 * 2932 * XXX: raise the limit to processing 2933 * at least 1 packet per tick. 2934 */ 2935 mac_srs->srs_bw->mac_bw_limit += 2936 mac_srs->srs_bw->mac_bw_limit; 2937 mac_srs->srs_bw->mac_bw_drop_threshold += 2938 mac_srs->srs_bw->mac_bw_drop_threshold; 2939 cmn_err(CE_NOTE, "mac_rx_srs_drain: srs(%p) " 2940 "raised B/W limit to %d since not even a " 2941 "single packet can be processed per " 2942 "tick %d\n", (void *)mac_srs, 2943 (int)mac_srs->srs_bw->mac_bw_limit, 2944 (int)msgdsize(mac_srs->srs_first)); 2945 } 2946 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2947 goto done; 2948 } 2949 2950 ASSERT(head != NULL); 2951 ASSERT(tail != NULL); 2952 2953 /* zero bandwidth: drop all and return to interrupt mode */ 2954 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2955 if (mac_srs->srs_bw->mac_bw_limit == 0) { 2956 srs_rx->sr_stat.mrs_sdrops += cnt; 2957 ASSERT(mac_srs->srs_bw->mac_bw_sz >= sz); 2958 mac_srs->srs_bw->mac_bw_sz -= sz; 2959 mac_srs->srs_bw->mac_bw_drop_bytes += sz; 2960 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2961 mac_drop_chain(head, "Rx no bandwidth"); 2962 goto leave_poll; 2963 } else { 2964 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2965 } 2966 2967 if ((tid = mac_srs->srs_tid) != NULL) 2968 mac_srs->srs_tid = NULL; 2969 2970 mac_srs->srs_state |= (SRS_PROC|proc_type); 2971 MAC_SRS_WORKER_POLLING_ON(mac_srs); 2972 2973 /* 2974 * mcip is NULL for broadcast and multicast flows. The promisc 2975 * callbacks for broadcast and multicast packets are delivered from 2976 * mac_rx() and we don't need to worry about that case in this path 2977 */ 2978 if (mcip != NULL) { 2979 if (mcip->mci_promisc_list != NULL) { 2980 mutex_exit(&mac_srs->srs_lock); 2981 mac_promisc_client_dispatch(mcip, head); 2982 mutex_enter(&mac_srs->srs_lock); 2983 } 2984 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) { 2985 mutex_exit(&mac_srs->srs_lock); 2986 mac_protect_intercept_dynamic(mcip, head); 2987 mutex_enter(&mac_srs->srs_lock); 2988 } 2989 } 2990 2991 /* 2992 * Check if SRS itself is doing the processing 2993 * This direct path does not apply when subflows are present. In this 2994 * case, packets need to be dispatched to a soft ring according to the 2995 * flow's bandwidth and other resources contraints. 2996 */ 2997 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) { 2998 mac_direct_rx_t proc; 2999 void *arg1; 3000 mac_resource_handle_t arg2; 3001 3002 /* 3003 * This is the case when a Rx is directly 3004 * assigned and we have a fully classified 3005 * protocol chain. We can deal with it in 3006 * one shot. 3007 */ 3008 proc = srs_rx->sr_func; 3009 arg1 = srs_rx->sr_arg1; 3010 arg2 = srs_rx->sr_arg2; 3011 3012 mac_srs->srs_state |= SRS_CLIENT_PROC; 3013 mutex_exit(&mac_srs->srs_lock); 3014 if (tid != NULL) { 3015 (void) untimeout(tid); 3016 tid = NULL; 3017 } 3018 3019 proc(arg1, arg2, head, NULL); 3020 /* 3021 * Decrement the size and count here itelf 3022 * since the packet has been processed. 3023 */ 3024 mutex_enter(&mac_srs->srs_lock); 3025 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 3026 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz); 3027 3028 if (mac_srs->srs_state & SRS_CLIENT_WAIT) 3029 cv_signal(&mac_srs->srs_client_cv); 3030 mac_srs->srs_state &= ~SRS_CLIENT_PROC; 3031 } else { 3032 /* Some kind of softrings based fanout is required */ 3033 mutex_exit(&mac_srs->srs_lock); 3034 if (tid != NULL) { 3035 (void) untimeout(tid); 3036 tid = NULL; 3037 } 3038 3039 /* 3040 * Since the fanout routines can deal with chains, 3041 * shoot the entire chain up. 3042 */ 3043 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP) 3044 mac_rx_srs_fanout(mac_srs, head); 3045 else 3046 mac_rx_srs_proto_fanout(mac_srs, head); 3047 mutex_enter(&mac_srs->srs_lock); 3048 } 3049 3050 /* 3051 * Send the poll thread to pick up any packets arrived 3052 * so far. This also serves as the last check in case 3053 * nothing else is queued in the SRS. The poll thread 3054 * is signalled only in the case the drain was done 3055 * by the worker thread and SRS_WORKER is set. The 3056 * worker thread can run in parallel as long as the 3057 * SRS_WORKER flag is set. We we have nothing else to 3058 * process, we can exit while leaving SRS_PROC set 3059 * which gives the poll thread control to process and 3060 * cleanup once it returns from the NIC. 3061 * 3062 * If we have nothing else to process, we need to 3063 * ensure that we keep holding the srs_lock till 3064 * all the checks below are done and control is 3065 * handed to the poll thread if it was running. 3066 */ 3067 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 3068 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 3069 if (mac_srs->srs_first != NULL) { 3070 if (proc_type == SRS_WORKER) { 3071 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3072 if (srs_rx->sr_poll_pkt_cnt <= 3073 srs_rx->sr_lowat) 3074 MAC_SRS_POLL_RING(mac_srs); 3075 goto again; 3076 } else { 3077 cv_signal(&mac_srs->srs_async); 3078 } 3079 } 3080 } 3081 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3082 3083 done: 3084 3085 if (mac_srs->srs_state & SRS_GET_PKTS) { 3086 /* 3087 * Poll thread is already running. Leave the 3088 * SRS_RPOC set and hand over the control to 3089 * poll thread. 3090 */ 3091 mac_srs->srs_state &= ~proc_type; 3092 return; 3093 } 3094 3095 /* 3096 * If we can't process packets because we have exceeded 3097 * B/W limit for this tick, just set the timeout 3098 * and leave. 3099 * 3100 * Even if there are no packets queued in SRS, we 3101 * need to make sure that the shared counter is 3102 * clear and any associated softrings have cleared 3103 * all the backlog. Otherwise, leave the interface 3104 * in polling mode and the poll thread will get 3105 * signalled once the count goes down to zero. 3106 * 3107 * If someone is already draining the queue (SRS_PROC is 3108 * set) when the srs_poll_pkt_cnt goes down to zero, 3109 * then it means that drain is already running and we 3110 * will turn off polling at that time if there is 3111 * no backlog. As long as there are packets queued either 3112 * is soft ring set or its soft rings, we will leave 3113 * the interface in polling mode. 3114 */ 3115 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 3116 if ((mac_srs->srs_state & SRS_POLLING_CAPAB) && 3117 ((mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) || 3118 (srs_rx->sr_poll_pkt_cnt > 0))) { 3119 MAC_SRS_POLLING_ON(mac_srs); 3120 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 3121 if ((mac_srs->srs_first != NULL) && 3122 (mac_srs->srs_tid == NULL)) 3123 mac_srs->srs_tid = timeout(mac_srs_fire, 3124 mac_srs, 1); 3125 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3126 return; 3127 } 3128 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3129 3130 leave_poll: 3131 3132 /* Nothing else to do. Get out of poll mode */ 3133 MAC_SRS_POLLING_OFF(mac_srs); 3134 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 3135 } 3136 3137 /* 3138 * mac_srs_worker 3139 * 3140 * The SRS worker routine. Drains the queue when no one else is 3141 * processing it. 3142 */ 3143 void 3144 mac_srs_worker(mac_soft_ring_set_t *mac_srs) 3145 { 3146 kmutex_t *lock = &mac_srs->srs_lock; 3147 kcondvar_t *async = &mac_srs->srs_async; 3148 callb_cpr_t cprinfo; 3149 boolean_t bw_ctl_flag; 3150 3151 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "srs_worker"); 3152 mutex_enter(lock); 3153 3154 start: 3155 for (;;) { 3156 bw_ctl_flag = B_FALSE; 3157 if (mac_srs->srs_type & SRST_BW_CONTROL) { 3158 MAC_SRS_BW_LOCK(mac_srs); 3159 MAC_SRS_CHECK_BW_CONTROL(mac_srs); 3160 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) 3161 bw_ctl_flag = B_TRUE; 3162 MAC_SRS_BW_UNLOCK(mac_srs); 3163 } 3164 /* 3165 * The SRS_BW_ENFORCED flag may change since we have dropped 3166 * the mac_bw_lock. However the drain function can handle both 3167 * a drainable SRS or a bandwidth controlled SRS, and the 3168 * effect of scheduling a timeout is to wakeup the worker 3169 * thread which in turn will call the drain function. Since 3170 * we release the srs_lock atomically only in the cv_wait there 3171 * isn't a fear of waiting for ever. 3172 */ 3173 while (((mac_srs->srs_state & SRS_PROC) || 3174 (mac_srs->srs_first == NULL) || bw_ctl_flag || 3175 (mac_srs->srs_state & SRS_TX_BLOCKED)) && 3176 !(mac_srs->srs_state & SRS_PAUSE)) { 3177 /* 3178 * If we have packets queued and we are here 3179 * because B/W control is in place, we better 3180 * schedule the worker wakeup after 1 tick 3181 * to see if bandwidth control can be relaxed. 3182 */ 3183 if (bw_ctl_flag && mac_srs->srs_tid == NULL) { 3184 /* 3185 * We need to ensure that a timer is already 3186 * scheduled or we force schedule one for 3187 * later so that we can continue processing 3188 * after this quanta is over. 3189 */ 3190 mac_srs->srs_tid = timeout(mac_srs_fire, 3191 mac_srs, 1); 3192 } 3193 wait: 3194 CALLB_CPR_SAFE_BEGIN(&cprinfo); 3195 cv_wait(async, lock); 3196 CALLB_CPR_SAFE_END(&cprinfo, lock); 3197 3198 if (mac_srs->srs_state & SRS_PAUSE) 3199 goto done; 3200 if (mac_srs->srs_state & SRS_PROC) 3201 goto wait; 3202 3203 if (mac_srs->srs_first != NULL && 3204 mac_srs->srs_type & SRST_BW_CONTROL) { 3205 MAC_SRS_BW_LOCK(mac_srs); 3206 if (mac_srs->srs_bw->mac_bw_state & 3207 SRS_BW_ENFORCED) { 3208 MAC_SRS_CHECK_BW_CONTROL(mac_srs); 3209 } 3210 bw_ctl_flag = mac_srs->srs_bw->mac_bw_state & 3211 SRS_BW_ENFORCED; 3212 MAC_SRS_BW_UNLOCK(mac_srs); 3213 } 3214 } 3215 3216 if (mac_srs->srs_state & SRS_PAUSE) 3217 goto done; 3218 mac_srs->srs_drain_func(mac_srs, SRS_WORKER); 3219 } 3220 done: 3221 /* 3222 * The Rx SRS quiesce logic first cuts off packet supply to the SRS 3223 * from both hard and soft classifications and waits for such threads 3224 * to finish before signaling the worker. So at this point the only 3225 * thread left that could be competing with the worker is the poll 3226 * thread. In the case of Tx, there shouldn't be any thread holding 3227 * SRS_PROC at this point. 3228 */ 3229 if (!(mac_srs->srs_state & SRS_PROC)) { 3230 mac_srs->srs_state |= SRS_PROC; 3231 } else { 3232 ASSERT((mac_srs->srs_type & SRST_TX) == 0); 3233 /* 3234 * Poll thread still owns the SRS and is still running 3235 */ 3236 ASSERT((mac_srs->srs_poll_thr == NULL) || 3237 ((mac_srs->srs_state & SRS_POLL_THR_OWNER) == 3238 SRS_POLL_THR_OWNER)); 3239 } 3240 mac_srs_worker_quiesce(mac_srs); 3241 /* 3242 * Wait for the SRS_RESTART or SRS_CONDEMNED signal from the initiator 3243 * of the quiesce operation 3244 */ 3245 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_RESTART))) 3246 cv_wait(&mac_srs->srs_async, &mac_srs->srs_lock); 3247 3248 if (mac_srs->srs_state & SRS_RESTART) { 3249 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED)); 3250 mac_srs_worker_restart(mac_srs); 3251 mac_srs->srs_state &= ~SRS_PROC; 3252 goto start; 3253 } 3254 3255 if (!(mac_srs->srs_state & SRS_CONDEMNED_DONE)) 3256 mac_srs_worker_quiesce(mac_srs); 3257 3258 mac_srs->srs_state &= ~SRS_PROC; 3259 /* The macro drops the srs_lock */ 3260 CALLB_CPR_EXIT(&cprinfo); 3261 thread_exit(); 3262 } 3263 3264 /* 3265 * mac_rx_srs_subflow_process 3266 * 3267 * Receive side routine called from interrupt path when there are 3268 * sub flows present on this SRS. 3269 */ 3270 /* ARGSUSED */ 3271 void 3272 mac_rx_srs_subflow_process(void *arg, mac_resource_handle_t srs, 3273 mblk_t *mp_chain, boolean_t loopback) 3274 { 3275 flow_entry_t *flent = NULL; 3276 flow_entry_t *prev_flent = NULL; 3277 mblk_t *mp = NULL; 3278 mblk_t *tail = NULL; 3279 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs; 3280 mac_client_impl_t *mcip; 3281 3282 mcip = mac_srs->srs_mcip; 3283 ASSERT(mcip != NULL); 3284 3285 /* 3286 * We need to determine the SRS for every packet 3287 * by walking the flow table, if we don't get any, 3288 * then we proceed using the SRS we came with. 3289 */ 3290 mp = tail = mp_chain; 3291 while (mp != NULL) { 3292 3293 /* 3294 * We will increment the stats for the matching subflow. 3295 * when we get the bytes/pkt count for the classified packets 3296 * later in mac_rx_srs_process. 3297 */ 3298 (void) mac_flow_lookup(mcip->mci_subflow_tab, mp, 3299 FLOW_INBOUND, &flent); 3300 3301 if (mp == mp_chain || flent == prev_flent) { 3302 if (prev_flent != NULL) 3303 FLOW_REFRELE(prev_flent); 3304 prev_flent = flent; 3305 flent = NULL; 3306 tail = mp; 3307 mp = mp->b_next; 3308 continue; 3309 } 3310 tail->b_next = NULL; 3311 /* 3312 * A null indicates, this is for the mac_srs itself. 3313 * XXX-venu : probably assert for fe_rx_srs_cnt == 0. 3314 */ 3315 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) { 3316 mac_rx_srs_process(arg, 3317 (mac_resource_handle_t)mac_srs, mp_chain, 3318 loopback); 3319 } else { 3320 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1, 3321 prev_flent->fe_cb_arg2, mp_chain, loopback); 3322 FLOW_REFRELE(prev_flent); 3323 } 3324 prev_flent = flent; 3325 flent = NULL; 3326 mp_chain = mp; 3327 tail = mp; 3328 mp = mp->b_next; 3329 } 3330 /* Last chain */ 3331 ASSERT(mp_chain != NULL); 3332 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) { 3333 mac_rx_srs_process(arg, 3334 (mac_resource_handle_t)mac_srs, mp_chain, loopback); 3335 } else { 3336 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1, 3337 prev_flent->fe_cb_arg2, mp_chain, loopback); 3338 FLOW_REFRELE(prev_flent); 3339 } 3340 } 3341 3342 /* 3343 * MAC SRS receive side routine. If the data is coming from the 3344 * network (i.e. from a NIC) then this is called in interrupt context. 3345 * If the data is coming from a local sender (e.g. mac_tx_send() or 3346 * bridge_forward()) then this is not called in interrupt context. 3347 * 3348 * loopback is set to force a context switch on the loopback 3349 * path between MAC clients. 3350 */ 3351 /* ARGSUSED */ 3352 void 3353 mac_rx_srs_process(void *arg, mac_resource_handle_t srs, mblk_t *mp_chain, 3354 boolean_t loopback) 3355 { 3356 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs; 3357 mblk_t *mp, *tail, *head; 3358 int count = 0; 3359 int count1; 3360 size_t sz = 0; 3361 size_t chain_sz, sz1; 3362 mac_bw_ctl_t *mac_bw; 3363 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 3364 3365 /* 3366 * Set the tail, count and sz. We set the sz irrespective 3367 * of whether we are doing B/W control or not for the 3368 * purpose of updating the stats. 3369 */ 3370 mp = tail = mp_chain; 3371 while (mp != NULL) { 3372 tail = mp; 3373 count++; 3374 sz += msgdsize(mp); 3375 mp = mp->b_next; 3376 } 3377 3378 mutex_enter(&mac_srs->srs_lock); 3379 3380 if (loopback) { 3381 SRS_RX_STAT_UPDATE(mac_srs, lclbytes, sz); 3382 SRS_RX_STAT_UPDATE(mac_srs, lclcnt, count); 3383 3384 } else { 3385 SRS_RX_STAT_UPDATE(mac_srs, intrbytes, sz); 3386 SRS_RX_STAT_UPDATE(mac_srs, intrcnt, count); 3387 } 3388 3389 /* 3390 * If the SRS in already being processed; has been blanked; 3391 * can be processed by worker thread only; or the B/W limit 3392 * has been reached, then queue the chain and check if 3393 * worker thread needs to be awakend. 3394 */ 3395 if (mac_srs->srs_type & SRST_BW_CONTROL) { 3396 mac_bw = mac_srs->srs_bw; 3397 ASSERT(mac_bw != NULL); 3398 mutex_enter(&mac_bw->mac_bw_lock); 3399 mac_bw->mac_bw_intr += sz; 3400 if (mac_bw->mac_bw_limit == 0) { 3401 /* zero bandwidth: drop all */ 3402 srs_rx->sr_stat.mrs_sdrops += count; 3403 mac_bw->mac_bw_drop_bytes += sz; 3404 mutex_exit(&mac_bw->mac_bw_lock); 3405 mutex_exit(&mac_srs->srs_lock); 3406 mac_drop_chain(mp_chain, "Rx no bandwidth"); 3407 return; 3408 } else { 3409 if ((mac_bw->mac_bw_sz + sz) <= 3410 mac_bw->mac_bw_drop_threshold) { 3411 mutex_exit(&mac_bw->mac_bw_lock); 3412 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, 3413 tail, count, sz); 3414 } else { 3415 mp = mp_chain; 3416 chain_sz = 0; 3417 count1 = 0; 3418 tail = NULL; 3419 head = NULL; 3420 while (mp != NULL) { 3421 sz1 = msgdsize(mp); 3422 if (mac_bw->mac_bw_sz + chain_sz + sz1 > 3423 mac_bw->mac_bw_drop_threshold) 3424 break; 3425 chain_sz += sz1; 3426 count1++; 3427 tail = mp; 3428 mp = mp->b_next; 3429 } 3430 mutex_exit(&mac_bw->mac_bw_lock); 3431 if (tail != NULL) { 3432 head = tail->b_next; 3433 tail->b_next = NULL; 3434 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, 3435 mp_chain, tail, count1, chain_sz); 3436 sz -= chain_sz; 3437 count -= count1; 3438 } else { 3439 /* Can't pick up any */ 3440 head = mp_chain; 3441 } 3442 if (head != NULL) { 3443 /* Drop any packet over the threshold */ 3444 srs_rx->sr_stat.mrs_sdrops += count; 3445 mutex_enter(&mac_bw->mac_bw_lock); 3446 mac_bw->mac_bw_drop_bytes += sz; 3447 mutex_exit(&mac_bw->mac_bw_lock); 3448 freemsgchain(head); 3449 } 3450 } 3451 MAC_SRS_WORKER_WAKEUP(mac_srs); 3452 mutex_exit(&mac_srs->srs_lock); 3453 return; 3454 } 3455 } 3456 3457 /* 3458 * If the total number of packets queued in the SRS and 3459 * its associated soft rings exceeds the max allowed, 3460 * then drop the chain. If we are polling capable, this 3461 * shouldn't be happening. 3462 */ 3463 if (!(mac_srs->srs_type & SRST_BW_CONTROL) && 3464 (srs_rx->sr_poll_pkt_cnt > srs_rx->sr_hiwat)) { 3465 mac_bw = mac_srs->srs_bw; 3466 srs_rx->sr_stat.mrs_sdrops += count; 3467 mutex_enter(&mac_bw->mac_bw_lock); 3468 mac_bw->mac_bw_drop_bytes += sz; 3469 mutex_exit(&mac_bw->mac_bw_lock); 3470 freemsgchain(mp_chain); 3471 mutex_exit(&mac_srs->srs_lock); 3472 return; 3473 } 3474 3475 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, tail, count, sz); 3476 3477 if (!(mac_srs->srs_state & SRS_PROC)) { 3478 /* 3479 * If we are coming via loopback, if we are not optimizing for 3480 * latency, or if our stack is running deep, we should signal 3481 * the worker thread. 3482 */ 3483 if (loopback || !(mac_srs->srs_state & SRS_LATENCY_OPT)) { 3484 /* 3485 * For loopback, We need to let the worker take 3486 * over as we don't want to continue in the same 3487 * thread even if we can. This could lead to stack 3488 * overflows and may also end up using 3489 * resources (cpu) incorrectly. 3490 */ 3491 cv_signal(&mac_srs->srs_async); 3492 } else if (STACK_BIAS + (uintptr_t)getfp() - 3493 (uintptr_t)curthread->t_stkbase < mac_rx_srs_stack_needed) { 3494 if (++mac_rx_srs_stack_toodeep == 0) 3495 mac_rx_srs_stack_toodeep = 1; 3496 cv_signal(&mac_srs->srs_async); 3497 } else { 3498 /* 3499 * Seems like no one is processing the SRS and 3500 * there is no backlog. We also inline process 3501 * our packet if its a single packet in non 3502 * latency optimized case (in latency optimized 3503 * case, we inline process chains of any size). 3504 */ 3505 mac_srs->srs_drain_func(mac_srs, SRS_PROC_FAST); 3506 } 3507 } 3508 mutex_exit(&mac_srs->srs_lock); 3509 } 3510 3511 /* TX SIDE ROUTINES (RUNTIME) */ 3512 3513 /* 3514 * mac_tx_srs_no_desc 3515 * 3516 * This routine is called by Tx single ring default mode 3517 * when Tx ring runs out of descs. 3518 */ 3519 mac_tx_cookie_t 3520 mac_tx_srs_no_desc(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3521 uint16_t flag, mblk_t **ret_mp) 3522 { 3523 mac_tx_cookie_t cookie = 0; 3524 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3525 boolean_t wakeup_worker = B_TRUE; 3526 uint32_t tx_mode = srs_tx->st_mode; 3527 int cnt, sz; 3528 mblk_t *tail; 3529 3530 ASSERT(tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_BW); 3531 if (flag & MAC_DROP_ON_NO_DESC) { 3532 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie, 3533 "Tx no desc"); 3534 } else { 3535 if (mac_srs->srs_first != NULL) 3536 wakeup_worker = B_FALSE; 3537 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3538 if (flag & MAC_TX_NO_ENQUEUE) { 3539 /* 3540 * If TX_QUEUED is not set, queue the 3541 * packet and let mac_tx_srs_drain() 3542 * set the TX_BLOCKED bit for the 3543 * reasons explained above. Otherwise, 3544 * return the mblks. 3545 */ 3546 if (wakeup_worker) { 3547 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3548 mp_chain, tail, cnt, sz); 3549 } else { 3550 MAC_TX_SET_NO_ENQUEUE(mac_srs, 3551 mp_chain, ret_mp, cookie); 3552 } 3553 } else { 3554 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain, 3555 tail, cnt, sz, cookie); 3556 } 3557 if (wakeup_worker) 3558 cv_signal(&mac_srs->srs_async); 3559 } 3560 return (cookie); 3561 } 3562 3563 /* 3564 * mac_tx_srs_enqueue 3565 * 3566 * This routine is called when Tx SRS is operating in either serializer 3567 * or bandwidth mode. In serializer mode, a packet will get enqueued 3568 * when a thread cannot enter SRS exclusively. In bandwidth mode, 3569 * packets gets queued if allowed byte-count limit for a tick is 3570 * exceeded. The action that gets taken when MAC_DROP_ON_NO_DESC and 3571 * MAC_TX_NO_ENQUEUE is set is different than when operaing in either 3572 * the default mode or fanout mode. Here packets get dropped or 3573 * returned back to the caller only after hi-watermark worth of data 3574 * is queued. 3575 */ 3576 static mac_tx_cookie_t 3577 mac_tx_srs_enqueue(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3578 uint16_t flag, uintptr_t fanout_hint, mblk_t **ret_mp) 3579 { 3580 mac_tx_cookie_t cookie = 0; 3581 int cnt, sz; 3582 mblk_t *tail; 3583 boolean_t wakeup_worker = B_TRUE; 3584 3585 /* 3586 * Ignore fanout hint if we don't have multiple tx rings. 3587 */ 3588 if (!MAC_TX_SOFT_RINGS(mac_srs)) 3589 fanout_hint = 0; 3590 3591 if (mac_srs->srs_first != NULL) 3592 wakeup_worker = B_FALSE; 3593 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3594 if (flag & MAC_DROP_ON_NO_DESC) { 3595 if (mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) { 3596 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie, 3597 "Tx SRS hiwat"); 3598 } else { 3599 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3600 mp_chain, tail, cnt, sz); 3601 } 3602 } else if (flag & MAC_TX_NO_ENQUEUE) { 3603 if ((mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) || 3604 (mac_srs->srs_state & SRS_TX_WAKEUP_CLIENT)) { 3605 MAC_TX_SET_NO_ENQUEUE(mac_srs, mp_chain, 3606 ret_mp, cookie); 3607 } else { 3608 mp_chain->b_prev = (mblk_t *)fanout_hint; 3609 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3610 mp_chain, tail, cnt, sz); 3611 } 3612 } else { 3613 /* 3614 * If you are BW_ENFORCED, just enqueue the 3615 * packet. srs_worker will drain it at the 3616 * prescribed rate. Before enqueueing, save 3617 * the fanout hint. 3618 */ 3619 mp_chain->b_prev = (mblk_t *)fanout_hint; 3620 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain, 3621 tail, cnt, sz, cookie); 3622 } 3623 if (wakeup_worker) 3624 cv_signal(&mac_srs->srs_async); 3625 return (cookie); 3626 } 3627 3628 /* 3629 * There are seven tx modes: 3630 * 3631 * 1) Default mode (SRS_TX_DEFAULT) 3632 * 2) Serialization mode (SRS_TX_SERIALIZE) 3633 * 3) Fanout mode (SRS_TX_FANOUT) 3634 * 4) Bandwdith mode (SRS_TX_BW) 3635 * 5) Fanout and Bandwidth mode (SRS_TX_BW_FANOUT) 3636 * 6) aggr Tx mode (SRS_TX_AGGR) 3637 * 7) aggr Tx bw mode (SRS_TX_BW_AGGR) 3638 * 3639 * The tx mode in which an SRS operates is decided in mac_tx_srs_setup() 3640 * based on the number of Tx rings requested for an SRS and whether 3641 * bandwidth control is requested or not. 3642 * 3643 * The default mode (i.e., no fanout/no bandwidth) is used when the 3644 * underlying NIC does not have Tx rings or just one Tx ring. In this mode, 3645 * the SRS acts as a pass-thru. Packets will go directly to mac_tx_send(). 3646 * When the underlying Tx ring runs out of Tx descs, it starts queueing up 3647 * packets in SRS. When flow-control is relieved, the srs_worker drains 3648 * the queued packets and informs blocked clients to restart sending 3649 * packets. 3650 * 3651 * In the SRS_TX_SERIALIZE mode, all calls to mac_tx() are serialized. This 3652 * mode is used when the link has no Tx rings or only one Tx ring. 3653 * 3654 * In the SRS_TX_FANOUT mode, packets will be fanned out to multiple 3655 * Tx rings. Each Tx ring will have a soft ring associated with it. 3656 * These soft rings will be hung off the Tx SRS. Queueing if it happens 3657 * due to lack of Tx desc will be in individual soft ring (and not srs) 3658 * associated with Tx ring. 3659 * 3660 * In the TX_BW mode, tx srs will allow packets to go down to Tx ring 3661 * only if bw is available. Otherwise the packets will be queued in 3662 * SRS. If fanout to multiple Tx rings is configured, the packets will 3663 * be fanned out among the soft rings associated with the Tx rings. 3664 * 3665 * In SRS_TX_AGGR mode, mac_tx_aggr_mode() routine is called. This routine 3666 * invokes an aggr function, aggr_find_tx_ring(), to find a pseudo Tx ring 3667 * belonging to a port on which the packet has to be sent. Aggr will 3668 * always have a pseudo Tx ring associated with it even when it is an 3669 * aggregation over a single NIC that has no Tx rings. Even in such a 3670 * case, the single pseudo Tx ring will have a soft ring associated with 3671 * it and the soft ring will hang off the SRS. 3672 * 3673 * If a bandwidth is specified for an aggr, SRS_TX_BW_AGGR mode is used. 3674 * In this mode, the bandwidth is first applied on the outgoing packets 3675 * and later mac_tx_addr_mode() function is called to send the packet out 3676 * of one of the pseudo Tx rings. 3677 * 3678 * Four flags are used in srs_state for indicating flow control 3679 * conditions : SRS_TX_BLOCKED, SRS_TX_HIWAT, SRS_TX_WAKEUP_CLIENT. 3680 * SRS_TX_BLOCKED indicates out of Tx descs. SRS expects a wakeup from the 3681 * driver below. 3682 * SRS_TX_HIWAT indicates packet count enqueued in Tx SRS exceeded Tx hiwat 3683 * and flow-control pressure is applied back to clients. The clients expect 3684 * wakeup when flow-control is relieved. 3685 * SRS_TX_WAKEUP_CLIENT get set when (flag == MAC_TX_NO_ENQUEUE) and mblk 3686 * got returned back to client either due to lack of Tx descs or due to bw 3687 * control reasons. The clients expect a wakeup when condition is relieved. 3688 * 3689 * The fourth argument to mac_tx() is the flag. Normally it will be 0 but 3690 * some clients set the following values too: MAC_DROP_ON_NO_DESC, 3691 * MAC_TX_NO_ENQUEUE 3692 * Mac clients that do not want packets to be enqueued in the mac layer set 3693 * MAC_DROP_ON_NO_DESC value. The packets won't be queued in the Tx SRS or 3694 * Tx soft rings but instead get dropped when the NIC runs out of desc. The 3695 * behaviour of this flag is different when the Tx is running in serializer 3696 * or bandwidth mode. Under these (Serializer, bandwidth) modes, the packet 3697 * get dropped when Tx high watermark is reached. 3698 * There are some mac clients like vsw, aggr that want the mblks to be 3699 * returned back to clients instead of being queued in Tx SRS (or Tx soft 3700 * rings) under flow-control (i.e., out of desc or exceeding bw limits) 3701 * conditions. These clients call mac_tx() with MAC_TX_NO_ENQUEUE flag set. 3702 * In the default and Tx fanout mode, the un-transmitted mblks will be 3703 * returned back to the clients when the driver runs out of Tx descs. 3704 * SRS_TX_WAKEUP_CLIENT (or S_RING_WAKEUP_CLIENT) will be set in SRS (or 3705 * soft ring) so that the clients can be woken up when Tx desc become 3706 * available. When running in serializer or bandwidth mode mode, 3707 * SRS_TX_WAKEUP_CLIENT will be set when tx hi-watermark is reached. 3708 */ 3709 3710 mac_tx_func_t 3711 mac_tx_get_func(uint32_t mode) 3712 { 3713 return (mac_tx_mode_list[mode].mac_tx_func); 3714 } 3715 3716 /* ARGSUSED */ 3717 static mac_tx_cookie_t 3718 mac_tx_single_ring_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3719 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3720 { 3721 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3722 mac_tx_stats_t stats; 3723 mac_tx_cookie_t cookie = 0; 3724 3725 ASSERT(srs_tx->st_mode == SRS_TX_DEFAULT); 3726 3727 /* Regular case with a single Tx ring */ 3728 /* 3729 * SRS_TX_BLOCKED is set when underlying NIC runs 3730 * out of Tx descs and messages start getting 3731 * queued. It won't get reset until 3732 * tx_srs_drain() completely drains out the 3733 * messages. 3734 */ 3735 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) { 3736 /* Tx descs/resources not available */ 3737 mutex_enter(&mac_srs->srs_lock); 3738 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) { 3739 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, 3740 flag, ret_mp); 3741 mutex_exit(&mac_srs->srs_lock); 3742 return (cookie); 3743 } 3744 /* 3745 * While we were computing mblk count, the 3746 * flow control condition got relieved. 3747 * Continue with the transmission. 3748 */ 3749 mutex_exit(&mac_srs->srs_lock); 3750 } 3751 3752 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 3753 mp_chain, &stats); 3754 3755 /* 3756 * Multiple threads could be here sending packets. 3757 * Under such conditions, it is not possible to 3758 * automically set SRS_TX_BLOCKED bit to indicate 3759 * out of tx desc condition. To atomically set 3760 * this, we queue the returned packet and do 3761 * the setting of SRS_TX_BLOCKED in 3762 * mac_tx_srs_drain(). 3763 */ 3764 if (mp_chain != NULL) { 3765 mutex_enter(&mac_srs->srs_lock); 3766 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, flag, ret_mp); 3767 mutex_exit(&mac_srs->srs_lock); 3768 return (cookie); 3769 } 3770 SRS_TX_STATS_UPDATE(mac_srs, &stats); 3771 3772 return (0); 3773 } 3774 3775 /* 3776 * mac_tx_serialize_mode 3777 * 3778 * This is an experimental mode implemented as per the request of PAE. 3779 * In this mode, all callers attempting to send a packet to the NIC 3780 * will get serialized. Only one thread at any time will access the 3781 * NIC to send the packet out. 3782 */ 3783 /* ARGSUSED */ 3784 static mac_tx_cookie_t 3785 mac_tx_serializer_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3786 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3787 { 3788 mac_tx_stats_t stats; 3789 mac_tx_cookie_t cookie = 0; 3790 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3791 3792 /* Single ring, serialize below */ 3793 ASSERT(srs_tx->st_mode == SRS_TX_SERIALIZE); 3794 mutex_enter(&mac_srs->srs_lock); 3795 if ((mac_srs->srs_first != NULL) || 3796 (mac_srs->srs_state & SRS_PROC)) { 3797 /* 3798 * In serialization mode, queue all packets until 3799 * TX_HIWAT is set. 3800 * If drop bit is set, drop if TX_HIWAT is set. 3801 * If no_enqueue is set, still enqueue until hiwat 3802 * is set and return mblks after TX_HIWAT is set. 3803 */ 3804 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, 3805 flag, 0, ret_mp); 3806 mutex_exit(&mac_srs->srs_lock); 3807 return (cookie); 3808 } 3809 /* 3810 * No packets queued, nothing on proc and no flow 3811 * control condition. Fast-path, ok. Do inline 3812 * processing. 3813 */ 3814 mac_srs->srs_state |= SRS_PROC; 3815 mutex_exit(&mac_srs->srs_lock); 3816 3817 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 3818 mp_chain, &stats); 3819 3820 mutex_enter(&mac_srs->srs_lock); 3821 mac_srs->srs_state &= ~SRS_PROC; 3822 if (mp_chain != NULL) { 3823 cookie = mac_tx_srs_enqueue(mac_srs, 3824 mp_chain, flag, 0, ret_mp); 3825 } 3826 if (mac_srs->srs_first != NULL) { 3827 /* 3828 * We processed inline our packet and a new 3829 * packet/s got queued while we were 3830 * processing. Wakeup srs worker 3831 */ 3832 cv_signal(&mac_srs->srs_async); 3833 } 3834 mutex_exit(&mac_srs->srs_lock); 3835 3836 if (cookie == 0) 3837 SRS_TX_STATS_UPDATE(mac_srs, &stats); 3838 3839 return (cookie); 3840 } 3841 3842 /* 3843 * mac_tx_fanout_mode 3844 * 3845 * In this mode, the SRS will have access to multiple Tx rings to send 3846 * the packet out. The fanout hint that is passed as an argument is 3847 * used to find an appropriate ring to fanout the traffic. Each Tx 3848 * ring, in turn, will have a soft ring associated with it. If a Tx 3849 * ring runs out of Tx desc's the returned packet will be queued in 3850 * the soft ring associated with that Tx ring. The srs itself will not 3851 * queue any packets. 3852 */ 3853 3854 #define MAC_TX_SOFT_RING_PROCESS(chain) { \ 3855 index = COMPUTE_INDEX(hash, mac_srs->srs_tx_ring_count), \ 3856 softring = mac_srs->srs_tx_soft_rings[index]; \ 3857 cookie = mac_tx_soft_ring_process(softring, chain, flag, ret_mp); \ 3858 DTRACE_PROBE2(tx__fanout, uint64_t, hash, uint_t, index); \ 3859 } 3860 3861 static mac_tx_cookie_t 3862 mac_tx_fanout_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3863 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3864 { 3865 mac_soft_ring_t *softring; 3866 uint64_t hash; 3867 uint_t index; 3868 mac_tx_cookie_t cookie = 0; 3869 3870 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT || 3871 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT); 3872 if (fanout_hint != 0) { 3873 /* 3874 * The hint is specified by the caller, simply pass the 3875 * whole chain to the soft ring. 3876 */ 3877 hash = HASH_HINT(fanout_hint); 3878 MAC_TX_SOFT_RING_PROCESS(mp_chain); 3879 } else { 3880 mblk_t *last_mp, *cur_mp, *sub_chain; 3881 uint64_t last_hash = 0; 3882 uint_t media = mac_srs->srs_mcip->mci_mip->mi_info.mi_media; 3883 3884 /* 3885 * Compute the hash from the contents (headers) of the 3886 * packets of the mblk chain. Split the chains into 3887 * subchains of the same conversation. 3888 * 3889 * Since there may be more than one ring used for 3890 * sub-chains of the same call, and since the caller 3891 * does not maintain per conversation state since it 3892 * passed a zero hint, unsent subchains will be 3893 * dropped. 3894 */ 3895 3896 flag |= MAC_DROP_ON_NO_DESC; 3897 ret_mp = NULL; 3898 3899 ASSERT(ret_mp == NULL); 3900 3901 sub_chain = NULL; 3902 last_mp = NULL; 3903 3904 for (cur_mp = mp_chain; cur_mp != NULL; 3905 cur_mp = cur_mp->b_next) { 3906 hash = mac_pkt_hash(media, cur_mp, MAC_PKT_HASH_L4, 3907 B_TRUE); 3908 if (last_hash != 0 && hash != last_hash) { 3909 /* 3910 * Starting a different subchain, send current 3911 * chain out. 3912 */ 3913 ASSERT(last_mp != NULL); 3914 last_mp->b_next = NULL; 3915 MAC_TX_SOFT_RING_PROCESS(sub_chain); 3916 sub_chain = NULL; 3917 } 3918 3919 /* add packet to subchain */ 3920 if (sub_chain == NULL) 3921 sub_chain = cur_mp; 3922 last_mp = cur_mp; 3923 last_hash = hash; 3924 } 3925 3926 if (sub_chain != NULL) { 3927 /* send last subchain */ 3928 ASSERT(last_mp != NULL); 3929 last_mp->b_next = NULL; 3930 MAC_TX_SOFT_RING_PROCESS(sub_chain); 3931 } 3932 3933 cookie = 0; 3934 } 3935 3936 return (cookie); 3937 } 3938 3939 /* 3940 * mac_tx_bw_mode 3941 * 3942 * In the bandwidth mode, Tx srs will allow packets to go down to Tx ring 3943 * only if bw is available. Otherwise the packets will be queued in 3944 * SRS. If the SRS has multiple Tx rings, then packets will get fanned 3945 * out to a Tx rings. 3946 */ 3947 static mac_tx_cookie_t 3948 mac_tx_bw_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3949 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3950 { 3951 int cnt, sz; 3952 mblk_t *tail; 3953 mac_tx_cookie_t cookie = 0; 3954 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3955 clock_t now; 3956 3957 ASSERT(TX_BANDWIDTH_MODE(mac_srs)); 3958 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL); 3959 mutex_enter(&mac_srs->srs_lock); 3960 if (mac_srs->srs_bw->mac_bw_limit == 0) { 3961 /* 3962 * zero bandwidth, no traffic is sent: drop the packets, 3963 * or return the whole chain if the caller requests all 3964 * unsent packets back. 3965 */ 3966 if (flag & MAC_TX_NO_ENQUEUE) { 3967 cookie = (mac_tx_cookie_t)mac_srs; 3968 *ret_mp = mp_chain; 3969 } else { 3970 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie, 3971 "Tx no bandwidth"); 3972 } 3973 mutex_exit(&mac_srs->srs_lock); 3974 return (cookie); 3975 } else if ((mac_srs->srs_first != NULL) || 3976 (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 3977 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag, 3978 fanout_hint, ret_mp); 3979 mutex_exit(&mac_srs->srs_lock); 3980 return (cookie); 3981 } 3982 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3983 now = ddi_get_lbolt(); 3984 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 3985 mac_srs->srs_bw->mac_bw_curr_time = now; 3986 mac_srs->srs_bw->mac_bw_used = 0; 3987 } else if (mac_srs->srs_bw->mac_bw_used > 3988 mac_srs->srs_bw->mac_bw_limit) { 3989 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 3990 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3991 mp_chain, tail, cnt, sz); 3992 /* 3993 * Wakeup worker thread. Note that worker 3994 * thread has to be woken up so that it 3995 * can fire up the timer to be woken up 3996 * on the next tick. Also once 3997 * BW_ENFORCED is set, it can only be 3998 * reset by srs_worker thread. Until then 3999 * all packets will get queued up in SRS 4000 * and hence this this code path won't be 4001 * entered until BW_ENFORCED is reset. 4002 */ 4003 cv_signal(&mac_srs->srs_async); 4004 mutex_exit(&mac_srs->srs_lock); 4005 return (cookie); 4006 } 4007 4008 mac_srs->srs_bw->mac_bw_used += sz; 4009 mutex_exit(&mac_srs->srs_lock); 4010 4011 if (srs_tx->st_mode == SRS_TX_BW_FANOUT) { 4012 mac_soft_ring_t *softring; 4013 uint_t indx, hash; 4014 4015 hash = HASH_HINT(fanout_hint); 4016 indx = COMPUTE_INDEX(hash, 4017 mac_srs->srs_tx_ring_count); 4018 softring = mac_srs->srs_tx_soft_rings[indx]; 4019 return (mac_tx_soft_ring_process(softring, mp_chain, flag, 4020 ret_mp)); 4021 } else if (srs_tx->st_mode == SRS_TX_BW_AGGR) { 4022 return (mac_tx_aggr_mode(mac_srs, mp_chain, 4023 fanout_hint, flag, ret_mp)); 4024 } else { 4025 mac_tx_stats_t stats; 4026 4027 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 4028 mp_chain, &stats); 4029 4030 if (mp_chain != NULL) { 4031 mutex_enter(&mac_srs->srs_lock); 4032 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 4033 if (mac_srs->srs_bw->mac_bw_used > sz) 4034 mac_srs->srs_bw->mac_bw_used -= sz; 4035 else 4036 mac_srs->srs_bw->mac_bw_used = 0; 4037 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag, 4038 fanout_hint, ret_mp); 4039 mutex_exit(&mac_srs->srs_lock); 4040 return (cookie); 4041 } 4042 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4043 4044 return (0); 4045 } 4046 } 4047 4048 /* 4049 * mac_tx_aggr_mode 4050 * 4051 * This routine invokes an aggr function, aggr_find_tx_ring(), to find 4052 * a (pseudo) Tx ring belonging to a port on which the packet has to 4053 * be sent. aggr_find_tx_ring() first finds the outgoing port based on 4054 * L2/L3/L4 policy and then uses the fanout_hint passed to it to pick 4055 * a Tx ring from the selected port. 4056 * 4057 * Note that a port can be deleted from the aggregation. In such a case, 4058 * the aggregation layer first separates the port from the rest of the 4059 * ports making sure that port (and thus any Tx rings associated with 4060 * it) won't get selected in the call to aggr_find_tx_ring() function. 4061 * Later calls are made to mac_group_rem_ring() passing pseudo Tx ring 4062 * handles one by one which in turn will quiesce the Tx SRS and remove 4063 * the soft ring associated with the pseudo Tx ring. Unlike Rx side 4064 * where a cookie is used to protect against mac_rx_ring() calls on 4065 * rings that have been removed, no such cookie is needed on the Tx 4066 * side as the pseudo Tx ring won't be available anymore to 4067 * aggr_find_tx_ring() once the port has been removed. 4068 */ 4069 static mac_tx_cookie_t 4070 mac_tx_aggr_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 4071 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 4072 { 4073 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 4074 mac_tx_ring_fn_t find_tx_ring_fn; 4075 mac_ring_handle_t ring = NULL; 4076 void *arg; 4077 mac_soft_ring_t *sringp; 4078 4079 find_tx_ring_fn = srs_tx->st_capab_aggr.mca_find_tx_ring_fn; 4080 arg = srs_tx->st_capab_aggr.mca_arg; 4081 if (find_tx_ring_fn(arg, mp_chain, fanout_hint, &ring) == NULL) 4082 return (0); 4083 sringp = srs_tx->st_soft_rings[((mac_ring_t *)ring)->mr_index]; 4084 return (mac_tx_soft_ring_process(sringp, mp_chain, flag, ret_mp)); 4085 } 4086 4087 void 4088 mac_tx_invoke_callbacks(mac_client_impl_t *mcip, mac_tx_cookie_t cookie) 4089 { 4090 mac_cb_t *mcb; 4091 mac_tx_notify_cb_t *mtnfp; 4092 4093 /* Wakeup callback registered clients */ 4094 MAC_CALLBACK_WALKER_INC(&mcip->mci_tx_notify_cb_info); 4095 for (mcb = mcip->mci_tx_notify_cb_list; mcb != NULL; 4096 mcb = mcb->mcb_nextp) { 4097 mtnfp = (mac_tx_notify_cb_t *)mcb->mcb_objp; 4098 mtnfp->mtnf_fn(mtnfp->mtnf_arg, cookie); 4099 } 4100 MAC_CALLBACK_WALKER_DCR(&mcip->mci_tx_notify_cb_info, 4101 &mcip->mci_tx_notify_cb_list); 4102 } 4103 4104 /* ARGSUSED */ 4105 void 4106 mac_tx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 4107 { 4108 mblk_t *head, *tail; 4109 size_t sz; 4110 uint32_t tx_mode; 4111 uint_t saved_pkt_count; 4112 mac_tx_stats_t stats; 4113 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 4114 clock_t now; 4115 4116 saved_pkt_count = 0; 4117 ASSERT(mutex_owned(&mac_srs->srs_lock)); 4118 ASSERT(!(mac_srs->srs_state & SRS_PROC)); 4119 4120 mac_srs->srs_state |= SRS_PROC; 4121 4122 tx_mode = srs_tx->st_mode; 4123 if (tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_SERIALIZE) { 4124 if (mac_srs->srs_first != NULL) { 4125 head = mac_srs->srs_first; 4126 tail = mac_srs->srs_last; 4127 saved_pkt_count = mac_srs->srs_count; 4128 mac_srs->srs_first = NULL; 4129 mac_srs->srs_last = NULL; 4130 mac_srs->srs_count = 0; 4131 mutex_exit(&mac_srs->srs_lock); 4132 4133 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 4134 head, &stats); 4135 4136 mutex_enter(&mac_srs->srs_lock); 4137 if (head != NULL) { 4138 /* Device out of tx desc, set block */ 4139 if (head->b_next == NULL) 4140 VERIFY(head == tail); 4141 tail->b_next = mac_srs->srs_first; 4142 mac_srs->srs_first = head; 4143 mac_srs->srs_count += 4144 (saved_pkt_count - stats.mts_opackets); 4145 if (mac_srs->srs_last == NULL) 4146 mac_srs->srs_last = tail; 4147 MAC_TX_SRS_BLOCK(mac_srs, head); 4148 } else { 4149 srs_tx->st_woken_up = B_FALSE; 4150 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4151 } 4152 } 4153 } else if (tx_mode == SRS_TX_BW) { 4154 /* 4155 * We are here because the timer fired and we have some data 4156 * to tranmit. Also mac_tx_srs_worker should have reset 4157 * SRS_BW_ENFORCED flag 4158 */ 4159 ASSERT(!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)); 4160 head = tail = mac_srs->srs_first; 4161 while (mac_srs->srs_first != NULL) { 4162 tail = mac_srs->srs_first; 4163 tail->b_prev = NULL; 4164 mac_srs->srs_first = tail->b_next; 4165 if (mac_srs->srs_first == NULL) 4166 mac_srs->srs_last = NULL; 4167 mac_srs->srs_count--; 4168 sz = msgdsize(tail); 4169 mac_srs->srs_size -= sz; 4170 saved_pkt_count++; 4171 MAC_TX_UPDATE_BW_INFO(mac_srs, sz); 4172 4173 if (mac_srs->srs_bw->mac_bw_used < 4174 mac_srs->srs_bw->mac_bw_limit) 4175 continue; 4176 4177 now = ddi_get_lbolt(); 4178 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 4179 mac_srs->srs_bw->mac_bw_curr_time = now; 4180 mac_srs->srs_bw->mac_bw_used = sz; 4181 continue; 4182 } 4183 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 4184 break; 4185 } 4186 4187 ASSERT((head == NULL && tail == NULL) || 4188 (head != NULL && tail != NULL)); 4189 if (tail != NULL) { 4190 tail->b_next = NULL; 4191 mutex_exit(&mac_srs->srs_lock); 4192 4193 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 4194 head, &stats); 4195 4196 mutex_enter(&mac_srs->srs_lock); 4197 if (head != NULL) { 4198 uint_t size_sent; 4199 4200 /* Device out of tx desc, set block */ 4201 if (head->b_next == NULL) 4202 VERIFY(head == tail); 4203 tail->b_next = mac_srs->srs_first; 4204 mac_srs->srs_first = head; 4205 mac_srs->srs_count += 4206 (saved_pkt_count - stats.mts_opackets); 4207 if (mac_srs->srs_last == NULL) 4208 mac_srs->srs_last = tail; 4209 size_sent = sz - stats.mts_obytes; 4210 mac_srs->srs_size += size_sent; 4211 mac_srs->srs_bw->mac_bw_sz += size_sent; 4212 if (mac_srs->srs_bw->mac_bw_used > size_sent) { 4213 mac_srs->srs_bw->mac_bw_used -= 4214 size_sent; 4215 } else { 4216 mac_srs->srs_bw->mac_bw_used = 0; 4217 } 4218 MAC_TX_SRS_BLOCK(mac_srs, head); 4219 } else { 4220 srs_tx->st_woken_up = B_FALSE; 4221 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4222 } 4223 } 4224 } else if (tx_mode == SRS_TX_BW_FANOUT || tx_mode == SRS_TX_BW_AGGR) { 4225 mblk_t *prev; 4226 uint64_t hint; 4227 4228 /* 4229 * We are here because the timer fired and we 4230 * have some quota to tranmit. 4231 */ 4232 prev = NULL; 4233 head = tail = mac_srs->srs_first; 4234 while (mac_srs->srs_first != NULL) { 4235 tail = mac_srs->srs_first; 4236 mac_srs->srs_first = tail->b_next; 4237 if (mac_srs->srs_first == NULL) 4238 mac_srs->srs_last = NULL; 4239 mac_srs->srs_count--; 4240 sz = msgdsize(tail); 4241 mac_srs->srs_size -= sz; 4242 mac_srs->srs_bw->mac_bw_used += sz; 4243 if (prev == NULL) 4244 hint = (ulong_t)tail->b_prev; 4245 if (hint != (ulong_t)tail->b_prev) { 4246 prev->b_next = NULL; 4247 mutex_exit(&mac_srs->srs_lock); 4248 TX_SRS_TO_SOFT_RING(mac_srs, head, hint); 4249 head = tail; 4250 hint = (ulong_t)tail->b_prev; 4251 mutex_enter(&mac_srs->srs_lock); 4252 } 4253 4254 prev = tail; 4255 tail->b_prev = NULL; 4256 if (mac_srs->srs_bw->mac_bw_used < 4257 mac_srs->srs_bw->mac_bw_limit) 4258 continue; 4259 4260 now = ddi_get_lbolt(); 4261 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 4262 mac_srs->srs_bw->mac_bw_curr_time = now; 4263 mac_srs->srs_bw->mac_bw_used = 0; 4264 continue; 4265 } 4266 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 4267 break; 4268 } 4269 ASSERT((head == NULL && tail == NULL) || 4270 (head != NULL && tail != NULL)); 4271 if (tail != NULL) { 4272 tail->b_next = NULL; 4273 mutex_exit(&mac_srs->srs_lock); 4274 TX_SRS_TO_SOFT_RING(mac_srs, head, hint); 4275 mutex_enter(&mac_srs->srs_lock); 4276 } 4277 } 4278 /* 4279 * SRS_TX_FANOUT case not considered here because packets 4280 * won't be queued in the SRS for this case. Packets will 4281 * be sent directly to soft rings underneath and if there 4282 * is any queueing at all, it would be in Tx side soft 4283 * rings. 4284 */ 4285 4286 /* 4287 * When srs_count becomes 0, reset SRS_TX_HIWAT and 4288 * SRS_TX_WAKEUP_CLIENT and wakeup registered clients. 4289 */ 4290 if (mac_srs->srs_count == 0 && (mac_srs->srs_state & 4291 (SRS_TX_HIWAT | SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED))) { 4292 mac_client_impl_t *mcip = mac_srs->srs_mcip; 4293 boolean_t wakeup_required = B_FALSE; 4294 4295 if (mac_srs->srs_state & 4296 (SRS_TX_HIWAT|SRS_TX_WAKEUP_CLIENT)) { 4297 wakeup_required = B_TRUE; 4298 } 4299 mac_srs->srs_state &= ~(SRS_TX_HIWAT | 4300 SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED); 4301 mutex_exit(&mac_srs->srs_lock); 4302 if (wakeup_required) { 4303 mac_tx_invoke_callbacks(mcip, (mac_tx_cookie_t)mac_srs); 4304 /* 4305 * If the client is not the primary MAC client, then we 4306 * need to send the notification to the clients upper 4307 * MAC, i.e. mci_upper_mip. 4308 */ 4309 mac_tx_notify(mcip->mci_upper_mip != NULL ? 4310 mcip->mci_upper_mip : mcip->mci_mip); 4311 } 4312 mutex_enter(&mac_srs->srs_lock); 4313 } 4314 mac_srs->srs_state &= ~SRS_PROC; 4315 } 4316 4317 /* 4318 * Given a packet, get the flow_entry that identifies the flow 4319 * to which that packet belongs. The flow_entry will contain 4320 * the transmit function to be used to send the packet. If the 4321 * function returns NULL, the packet should be sent using the 4322 * underlying NIC. 4323 */ 4324 static flow_entry_t * 4325 mac_tx_classify(mac_impl_t *mip, mblk_t *mp) 4326 { 4327 flow_entry_t *flent = NULL; 4328 mac_client_impl_t *mcip; 4329 int err; 4330 4331 /* 4332 * Do classification on the packet. 4333 */ 4334 err = mac_flow_lookup(mip->mi_flow_tab, mp, FLOW_OUTBOUND, &flent); 4335 if (err != 0) 4336 return (NULL); 4337 4338 /* 4339 * This flent might just be an additional one on the MAC client, 4340 * i.e. for classification purposes (different fdesc), however 4341 * the resources, SRS et. al., are in the mci_flent, so if 4342 * this isn't the mci_flent, we need to get it. 4343 */ 4344 if ((mcip = flent->fe_mcip) != NULL && mcip->mci_flent != flent) { 4345 FLOW_REFRELE(flent); 4346 flent = mcip->mci_flent; 4347 FLOW_TRY_REFHOLD(flent, err); 4348 if (err != 0) 4349 return (NULL); 4350 } 4351 4352 return (flent); 4353 } 4354 4355 /* 4356 * This macro is only meant to be used by mac_tx_send(). 4357 */ 4358 #define CHECK_VID_AND_ADD_TAG(mp) { \ 4359 if (vid_check) { \ 4360 int err = 0; \ 4361 \ 4362 MAC_VID_CHECK(src_mcip, (mp), err); \ 4363 if (err != 0) { \ 4364 freemsg((mp)); \ 4365 (mp) = next; \ 4366 oerrors++; \ 4367 continue; \ 4368 } \ 4369 } \ 4370 if (add_tag) { \ 4371 (mp) = mac_add_vlan_tag((mp), 0, vid); \ 4372 if ((mp) == NULL) { \ 4373 (mp) = next; \ 4374 oerrors++; \ 4375 continue; \ 4376 } \ 4377 } \ 4378 } 4379 4380 mblk_t * 4381 mac_tx_send(mac_client_handle_t mch, mac_ring_handle_t ring, mblk_t *mp_chain, 4382 mac_tx_stats_t *stats) 4383 { 4384 mac_client_impl_t *src_mcip = (mac_client_impl_t *)mch; 4385 mac_impl_t *mip = src_mcip->mci_mip; 4386 uint_t obytes = 0, opackets = 0, oerrors = 0; 4387 mblk_t *mp = NULL, *next; 4388 boolean_t vid_check, add_tag; 4389 uint16_t vid = 0; 4390 4391 if (mip->mi_nclients > 1) { 4392 vid_check = MAC_VID_CHECK_NEEDED(src_mcip); 4393 add_tag = MAC_TAG_NEEDED(src_mcip); 4394 if (add_tag) 4395 vid = mac_client_vid(mch); 4396 } else { 4397 ASSERT(mip->mi_nclients == 1); 4398 vid_check = add_tag = B_FALSE; 4399 } 4400 4401 /* 4402 * Fastpath: if there's only one client, we simply send 4403 * the packet down to the underlying NIC. 4404 */ 4405 if (mip->mi_nactiveclients == 1) { 4406 DTRACE_PROBE2(fastpath, 4407 mac_client_impl_t *, src_mcip, mblk_t *, mp_chain); 4408 4409 mp = mp_chain; 4410 while (mp != NULL) { 4411 next = mp->b_next; 4412 mp->b_next = NULL; 4413 opackets++; 4414 obytes += (mp->b_cont == NULL ? MBLKL(mp) : 4415 msgdsize(mp)); 4416 4417 CHECK_VID_AND_ADD_TAG(mp); 4418 mp = mac_provider_tx(mip, ring, mp, src_mcip); 4419 4420 /* 4421 * If the driver is out of descriptors and does a 4422 * partial send it will return a chain of unsent 4423 * mblks. Adjust the accounting stats. 4424 */ 4425 if (mp != NULL) { 4426 opackets--; 4427 obytes -= msgdsize(mp); 4428 mp->b_next = next; 4429 break; 4430 } 4431 mp = next; 4432 } 4433 goto done; 4434 } 4435 4436 /* 4437 * No fastpath, we either have more than one MAC client 4438 * defined on top of the same MAC, or one or more MAC 4439 * client promiscuous callbacks. 4440 */ 4441 DTRACE_PROBE3(slowpath, mac_client_impl_t *, 4442 src_mcip, int, mip->mi_nclients, mblk_t *, mp_chain); 4443 4444 mp = mp_chain; 4445 while (mp != NULL) { 4446 flow_entry_t *dst_flow_ent; 4447 void *flow_cookie; 4448 size_t pkt_size; 4449 4450 next = mp->b_next; 4451 mp->b_next = NULL; 4452 opackets++; 4453 pkt_size = (mp->b_cont == NULL ? MBLKL(mp) : msgdsize(mp)); 4454 obytes += pkt_size; 4455 CHECK_VID_AND_ADD_TAG(mp); 4456 4457 /* 4458 * Find the destination. 4459 */ 4460 dst_flow_ent = mac_tx_classify(mip, mp); 4461 4462 if (dst_flow_ent != NULL) { 4463 /* 4464 * Got a matching flow. It's either another 4465 * MAC client, or a broadcast/multicast flow. 4466 */ 4467 flow_cookie = mac_flow_get_client_cookie(dst_flow_ent); 4468 4469 if (flow_cookie != NULL) { 4470 /* 4471 * The vnic_bcast_send function expects 4472 * to receive the sender MAC client 4473 * as value for arg2. 4474 */ 4475 mac_bcast_send(flow_cookie, src_mcip, mp, 4476 B_TRUE); 4477 } else { 4478 /* 4479 * loopback the packet to a local MAC 4480 * client. We force a context switch 4481 * if both source and destination MAC 4482 * clients are used by IP, i.e. 4483 * bypass is set. 4484 */ 4485 boolean_t do_switch; 4486 4487 mac_client_impl_t *dst_mcip = 4488 dst_flow_ent->fe_mcip; 4489 4490 /* 4491 * Check if there are promiscuous mode 4492 * callbacks defined. This check is 4493 * done here in the 'else' case and 4494 * not in other cases because this 4495 * path is for local loopback 4496 * communication which does not go 4497 * through MAC_TX(). For paths that go 4498 * through MAC_TX(), the promisc_list 4499 * check is done inside the MAC_TX() 4500 * macro. 4501 */ 4502 if (mip->mi_promisc_list != NULL) { 4503 mac_promisc_dispatch(mip, mp, src_mcip, 4504 B_TRUE); 4505 } 4506 4507 do_switch = ((src_mcip->mci_state_flags & 4508 dst_mcip->mci_state_flags & 4509 MCIS_CLIENT_POLL_CAPABLE) != 0); 4510 4511 mac_hw_emul(&mp, NULL, NULL, MAC_ALL_EMULS); 4512 if (mp != NULL) { 4513 (dst_flow_ent->fe_cb_fn)( 4514 dst_flow_ent->fe_cb_arg1, 4515 dst_flow_ent->fe_cb_arg2, 4516 mp, do_switch); 4517 } 4518 4519 } 4520 FLOW_REFRELE(dst_flow_ent); 4521 } else { 4522 /* 4523 * Unknown destination, send via the underlying 4524 * NIC. 4525 */ 4526 mp = mac_provider_tx(mip, ring, mp, src_mcip); 4527 if (mp != NULL) { 4528 /* 4529 * Adjust for the last packet that 4530 * could not be transmitted 4531 */ 4532 opackets--; 4533 obytes -= pkt_size; 4534 mp->b_next = next; 4535 break; 4536 } 4537 } 4538 mp = next; 4539 } 4540 4541 done: 4542 stats->mts_obytes = obytes; 4543 stats->mts_opackets = opackets; 4544 stats->mts_oerrors = oerrors; 4545 return (mp); 4546 } 4547 4548 /* 4549 * mac_tx_srs_ring_present 4550 * 4551 * Returns whether the specified ring is part of the specified SRS. 4552 */ 4553 boolean_t 4554 mac_tx_srs_ring_present(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring) 4555 { 4556 int i; 4557 mac_soft_ring_t *soft_ring; 4558 4559 if (srs->srs_tx.st_arg2 == tx_ring) 4560 return (B_TRUE); 4561 4562 for (i = 0; i < srs->srs_tx_ring_count; i++) { 4563 soft_ring = srs->srs_tx_soft_rings[i]; 4564 if (soft_ring->s_ring_tx_arg2 == tx_ring) 4565 return (B_TRUE); 4566 } 4567 4568 return (B_FALSE); 4569 } 4570 4571 /* 4572 * mac_tx_srs_get_soft_ring 4573 * 4574 * Returns the TX soft ring associated with the given ring, if present. 4575 */ 4576 mac_soft_ring_t * 4577 mac_tx_srs_get_soft_ring(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring) 4578 { 4579 int i; 4580 mac_soft_ring_t *soft_ring; 4581 4582 if (srs->srs_tx.st_arg2 == tx_ring) 4583 return (NULL); 4584 4585 for (i = 0; i < srs->srs_tx_ring_count; i++) { 4586 soft_ring = srs->srs_tx_soft_rings[i]; 4587 if (soft_ring->s_ring_tx_arg2 == tx_ring) 4588 return (soft_ring); 4589 } 4590 4591 return (NULL); 4592 } 4593 4594 /* 4595 * mac_tx_srs_wakeup 4596 * 4597 * Called when Tx desc become available. Wakeup the appropriate worker 4598 * thread after resetting the SRS_TX_BLOCKED/S_RING_BLOCK bit in the 4599 * state field. 4600 */ 4601 void 4602 mac_tx_srs_wakeup(mac_soft_ring_set_t *mac_srs, mac_ring_handle_t ring) 4603 { 4604 int i; 4605 mac_soft_ring_t *sringp; 4606 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 4607 4608 mutex_enter(&mac_srs->srs_lock); 4609 /* 4610 * srs_tx_ring_count == 0 is the single ring mode case. In 4611 * this mode, there will not be Tx soft rings associated 4612 * with the SRS. 4613 */ 4614 if (!MAC_TX_SOFT_RINGS(mac_srs)) { 4615 if (srs_tx->st_arg2 == ring && 4616 mac_srs->srs_state & SRS_TX_BLOCKED) { 4617 mac_srs->srs_state &= ~SRS_TX_BLOCKED; 4618 srs_tx->st_stat.mts_unblockcnt++; 4619 cv_signal(&mac_srs->srs_async); 4620 } 4621 /* 4622 * A wakeup can come before tx_srs_drain() could 4623 * grab srs lock and set SRS_TX_BLOCKED. So 4624 * always set woken_up flag when we come here. 4625 */ 4626 srs_tx->st_woken_up = B_TRUE; 4627 mutex_exit(&mac_srs->srs_lock); 4628 return; 4629 } 4630 4631 /* 4632 * If you are here, it is for FANOUT, BW_FANOUT, 4633 * AGGR_MODE or AGGR_BW_MODE case 4634 */ 4635 for (i = 0; i < mac_srs->srs_tx_ring_count; i++) { 4636 sringp = mac_srs->srs_tx_soft_rings[i]; 4637 mutex_enter(&sringp->s_ring_lock); 4638 if (sringp->s_ring_tx_arg2 == ring) { 4639 if (sringp->s_ring_state & S_RING_BLOCK) { 4640 sringp->s_ring_state &= ~S_RING_BLOCK; 4641 sringp->s_st_stat.mts_unblockcnt++; 4642 cv_signal(&sringp->s_ring_async); 4643 } 4644 sringp->s_ring_tx_woken_up = B_TRUE; 4645 } 4646 mutex_exit(&sringp->s_ring_lock); 4647 } 4648 mutex_exit(&mac_srs->srs_lock); 4649 } 4650 4651 /* 4652 * Once the driver is done draining, send a MAC_NOTE_TX notification to unleash 4653 * the blocked clients again. 4654 */ 4655 void 4656 mac_tx_notify(mac_impl_t *mip) 4657 { 4658 i_mac_notify(mip, MAC_NOTE_TX); 4659 } 4660 4661 /* 4662 * RX SOFTRING RELATED FUNCTIONS 4663 * 4664 * These functions really belong in mac_soft_ring.c and here for 4665 * a short period. 4666 */ 4667 4668 #define SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \ 4669 /* \ 4670 * Enqueue our mblk chain. \ 4671 */ \ 4672 ASSERT(MUTEX_HELD(&(ringp)->s_ring_lock)); \ 4673 \ 4674 if ((ringp)->s_ring_last != NULL) \ 4675 (ringp)->s_ring_last->b_next = (mp); \ 4676 else \ 4677 (ringp)->s_ring_first = (mp); \ 4678 (ringp)->s_ring_last = (tail); \ 4679 (ringp)->s_ring_count += (cnt); \ 4680 ASSERT((ringp)->s_ring_count > 0); \ 4681 if ((ringp)->s_ring_type & ST_RING_BW_CTL) { \ 4682 (ringp)->s_ring_size += sz; \ 4683 } \ 4684 } 4685 4686 /* 4687 * Default entry point to deliver a packet chain to a MAC client. 4688 * If the MAC client has flows, do the classification with these 4689 * flows as well. 4690 */ 4691 /* ARGSUSED */ 4692 void 4693 mac_rx_deliver(void *arg1, mac_resource_handle_t mrh, mblk_t *mp_chain, 4694 mac_header_info_t *arg3) 4695 { 4696 mac_client_impl_t *mcip = arg1; 4697 4698 if (mcip->mci_nvids == 1 && 4699 !(mcip->mci_state_flags & MCIS_STRIP_DISABLE)) { 4700 /* 4701 * If the client has exactly one VID associated with it 4702 * and striping of VLAN header is not disabled, 4703 * remove the VLAN tag from the packet before 4704 * passing it on to the client's receive callback. 4705 * Note that this needs to be done after we dispatch 4706 * the packet to the promiscuous listeners of the 4707 * client, since they expect to see the whole 4708 * frame including the VLAN headers. 4709 * 4710 * The MCIS_STRIP_DISABLE is only issued when sun4v 4711 * vsw is in play. 4712 */ 4713 mp_chain = mac_strip_vlan_tag_chain(mp_chain); 4714 } 4715 4716 mcip->mci_rx_fn(mcip->mci_rx_arg, mrh, mp_chain, B_FALSE); 4717 } 4718 4719 /* 4720 * Process a chain for a given soft ring. If the number of packets 4721 * queued in the SRS and its associated soft rings (including this 4722 * one) is very small (tracked by srs_poll_pkt_cnt) then allow the 4723 * entering thread (interrupt or poll thread) to process the chain 4724 * inline. This is meant to reduce latency under low load. 4725 * 4726 * The proc and arg for each mblk is already stored in the mblk in 4727 * appropriate places. 4728 */ 4729 /* ARGSUSED */ 4730 void 4731 mac_rx_soft_ring_process(mac_client_impl_t *mcip, mac_soft_ring_t *ringp, 4732 mblk_t *mp_chain, mblk_t *tail, int cnt, size_t sz) 4733 { 4734 mac_direct_rx_t proc; 4735 void *arg1; 4736 mac_resource_handle_t arg2; 4737 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4738 4739 ASSERT(ringp != NULL); 4740 ASSERT(mp_chain != NULL); 4741 ASSERT(tail != NULL); 4742 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4743 4744 mutex_enter(&ringp->s_ring_lock); 4745 ringp->s_ring_total_inpkt += cnt; 4746 ringp->s_ring_total_rbytes += sz; 4747 if ((mac_srs->srs_rx.sr_poll_pkt_cnt <= 1) && 4748 !(ringp->s_ring_type & ST_RING_WORKER_ONLY)) { 4749 /* If on processor or blanking on, then enqueue and return */ 4750 if (ringp->s_ring_state & S_RING_BLANK || 4751 ringp->s_ring_state & S_RING_PROC) { 4752 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4753 mutex_exit(&ringp->s_ring_lock); 4754 return; 4755 } 4756 proc = ringp->s_ring_rx_func; 4757 arg1 = ringp->s_ring_rx_arg1; 4758 arg2 = ringp->s_ring_rx_arg2; 4759 /* 4760 * See if anything is already queued. If we are the 4761 * first packet, do inline processing else queue the 4762 * packet and do the drain. 4763 */ 4764 if (ringp->s_ring_first == NULL) { 4765 /* 4766 * Fast-path, ok to process and nothing queued. 4767 */ 4768 ringp->s_ring_run = curthread; 4769 ringp->s_ring_state |= (S_RING_PROC); 4770 4771 mutex_exit(&ringp->s_ring_lock); 4772 4773 /* 4774 * We are the chain of 1 packet so 4775 * go through this fast path. 4776 */ 4777 ASSERT(mp_chain->b_next == NULL); 4778 4779 (*proc)(arg1, arg2, mp_chain, NULL); 4780 4781 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4782 /* 4783 * If we have an SRS performing bandwidth 4784 * control then we need to decrement the size 4785 * and count so the SRS has an accurate count 4786 * of the data queued between the SRS and its 4787 * soft rings. We decrement the counters only 4788 * when the packet is processed by both the 4789 * SRS and the soft ring. 4790 */ 4791 mutex_enter(&mac_srs->srs_lock); 4792 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 4793 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz); 4794 mutex_exit(&mac_srs->srs_lock); 4795 4796 mutex_enter(&ringp->s_ring_lock); 4797 ringp->s_ring_run = NULL; 4798 ringp->s_ring_state &= ~S_RING_PROC; 4799 if (ringp->s_ring_state & S_RING_CLIENT_WAIT) 4800 cv_signal(&ringp->s_ring_client_cv); 4801 4802 if ((ringp->s_ring_first == NULL) || 4803 (ringp->s_ring_state & S_RING_BLANK)) { 4804 /* 4805 * We processed a single packet inline 4806 * and nothing new has arrived or our 4807 * receiver doesn't want to receive 4808 * any packets. We are done. 4809 */ 4810 mutex_exit(&ringp->s_ring_lock); 4811 return; 4812 } 4813 } else { 4814 SOFT_RING_ENQUEUE_CHAIN(ringp, 4815 mp_chain, tail, cnt, sz); 4816 } 4817 4818 /* 4819 * We are here because either we couldn't do inline 4820 * processing (because something was already 4821 * queued), or we had a chain of more than one 4822 * packet, or something else arrived after we were 4823 * done with inline processing. 4824 */ 4825 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); 4826 ASSERT(ringp->s_ring_first != NULL); 4827 4828 ringp->s_ring_drain_func(ringp); 4829 mutex_exit(&ringp->s_ring_lock); 4830 return; 4831 } else { 4832 /* ST_RING_WORKER_ONLY case */ 4833 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4834 mac_soft_ring_worker_wakeup(ringp); 4835 mutex_exit(&ringp->s_ring_lock); 4836 } 4837 } 4838 4839 /* 4840 * TX SOFTRING RELATED FUNCTIONS 4841 * 4842 * These functions really belong in mac_soft_ring.c and here for 4843 * a short period. 4844 */ 4845 4846 #define TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \ 4847 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); \ 4848 ringp->s_ring_state |= S_RING_ENQUEUED; \ 4849 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); \ 4850 } 4851 4852 /* 4853 * mac_tx_sring_queued 4854 * 4855 * When we are out of transmit descriptors and we already have a 4856 * queue that exceeds hiwat (or the client called us with 4857 * MAC_TX_NO_ENQUEUE or MAC_DROP_ON_NO_DESC flag), return the 4858 * soft ring pointer as the opaque cookie for the client enable 4859 * flow control. 4860 */ 4861 static mac_tx_cookie_t 4862 mac_tx_sring_enqueue(mac_soft_ring_t *ringp, mblk_t *mp_chain, uint16_t flag, 4863 mblk_t **ret_mp) 4864 { 4865 int cnt; 4866 size_t sz; 4867 mblk_t *tail; 4868 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4869 mac_tx_cookie_t cookie = 0; 4870 boolean_t wakeup_worker = B_TRUE; 4871 4872 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); 4873 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 4874 if (flag & MAC_DROP_ON_NO_DESC) { 4875 mac_drop_chain(mp_chain, "Tx softring no desc"); 4876 /* increment freed stats */ 4877 ringp->s_ring_drops += cnt; 4878 cookie = (mac_tx_cookie_t)ringp; 4879 } else { 4880 if (ringp->s_ring_first != NULL) 4881 wakeup_worker = B_FALSE; 4882 4883 if (flag & MAC_TX_NO_ENQUEUE) { 4884 /* 4885 * If QUEUED is not set, queue the packet 4886 * and let mac_tx_soft_ring_drain() set 4887 * the TX_BLOCKED bit for the reasons 4888 * explained above. Otherwise, return the 4889 * mblks. 4890 */ 4891 if (wakeup_worker) { 4892 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, 4893 mp_chain, tail, cnt, sz); 4894 } else { 4895 ringp->s_ring_state |= S_RING_WAKEUP_CLIENT; 4896 cookie = (mac_tx_cookie_t)ringp; 4897 *ret_mp = mp_chain; 4898 } 4899 } else { 4900 boolean_t enqueue = B_TRUE; 4901 4902 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) { 4903 /* 4904 * flow-controlled. Store ringp in cookie 4905 * so that it can be returned as 4906 * mac_tx_cookie_t to client 4907 */ 4908 ringp->s_ring_state |= S_RING_TX_HIWAT; 4909 cookie = (mac_tx_cookie_t)ringp; 4910 ringp->s_ring_hiwat_cnt++; 4911 if (ringp->s_ring_count > 4912 ringp->s_ring_tx_max_q_cnt) { 4913 /* increment freed stats */ 4914 ringp->s_ring_drops += cnt; 4915 /* 4916 * b_prev may be set to the fanout hint 4917 * hence can't use freemsg directly 4918 */ 4919 mac_drop_chain(mp_chain, 4920 "Tx softring max queue"); 4921 DTRACE_PROBE1(tx_queued_hiwat, 4922 mac_soft_ring_t *, ringp); 4923 enqueue = B_FALSE; 4924 } 4925 } 4926 if (enqueue) { 4927 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, 4928 tail, cnt, sz); 4929 } 4930 } 4931 if (wakeup_worker) 4932 cv_signal(&ringp->s_ring_async); 4933 } 4934 return (cookie); 4935 } 4936 4937 4938 /* 4939 * mac_tx_soft_ring_process 4940 * 4941 * This routine is called when fanning out outgoing traffic among 4942 * multipe Tx rings. 4943 * Note that a soft ring is associated with a h/w Tx ring. 4944 */ 4945 mac_tx_cookie_t 4946 mac_tx_soft_ring_process(mac_soft_ring_t *ringp, mblk_t *mp_chain, 4947 uint16_t flag, mblk_t **ret_mp) 4948 { 4949 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4950 int cnt; 4951 size_t sz; 4952 mblk_t *tail; 4953 mac_tx_cookie_t cookie = 0; 4954 4955 ASSERT(ringp != NULL); 4956 ASSERT(mp_chain != NULL); 4957 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4958 /* 4959 * The following modes can come here: SRS_TX_BW_FANOUT, 4960 * SRS_TX_FANOUT, SRS_TX_AGGR, SRS_TX_BW_AGGR. 4961 */ 4962 ASSERT(MAC_TX_SOFT_RINGS(mac_srs)); 4963 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT || 4964 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT || 4965 mac_srs->srs_tx.st_mode == SRS_TX_AGGR || 4966 mac_srs->srs_tx.st_mode == SRS_TX_BW_AGGR); 4967 4968 if (ringp->s_ring_type & ST_RING_WORKER_ONLY) { 4969 /* Serialization mode */ 4970 4971 mutex_enter(&ringp->s_ring_lock); 4972 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) { 4973 cookie = mac_tx_sring_enqueue(ringp, mp_chain, 4974 flag, ret_mp); 4975 mutex_exit(&ringp->s_ring_lock); 4976 return (cookie); 4977 } 4978 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 4979 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4980 if (ringp->s_ring_state & (S_RING_BLOCK | S_RING_PROC)) { 4981 /* 4982 * If ring is blocked due to lack of Tx 4983 * descs, just return. Worker thread 4984 * will get scheduled when Tx desc's 4985 * become available. 4986 */ 4987 mutex_exit(&ringp->s_ring_lock); 4988 return (cookie); 4989 } 4990 mac_soft_ring_worker_wakeup(ringp); 4991 mutex_exit(&ringp->s_ring_lock); 4992 return (cookie); 4993 } else { 4994 /* Default fanout mode */ 4995 /* 4996 * S_RING_BLOCKED is set when underlying NIC runs 4997 * out of Tx descs and messages start getting 4998 * queued. It won't get reset until 4999 * tx_srs_drain() completely drains out the 5000 * messages. 5001 */ 5002 mac_tx_stats_t stats; 5003 5004 if (ringp->s_ring_state & S_RING_ENQUEUED) { 5005 /* Tx descs/resources not available */ 5006 mutex_enter(&ringp->s_ring_lock); 5007 if (ringp->s_ring_state & S_RING_ENQUEUED) { 5008 cookie = mac_tx_sring_enqueue(ringp, mp_chain, 5009 flag, ret_mp); 5010 mutex_exit(&ringp->s_ring_lock); 5011 return (cookie); 5012 } 5013 /* 5014 * While we were computing mblk count, the 5015 * flow control condition got relieved. 5016 * Continue with the transmission. 5017 */ 5018 mutex_exit(&ringp->s_ring_lock); 5019 } 5020 5021 mp_chain = mac_tx_send(ringp->s_ring_tx_arg1, 5022 ringp->s_ring_tx_arg2, mp_chain, &stats); 5023 5024 /* 5025 * Multiple threads could be here sending packets. 5026 * Under such conditions, it is not possible to 5027 * automically set S_RING_BLOCKED bit to indicate 5028 * out of tx desc condition. To atomically set 5029 * this, we queue the returned packet and do 5030 * the setting of S_RING_BLOCKED in 5031 * mac_tx_soft_ring_drain(). 5032 */ 5033 if (mp_chain != NULL) { 5034 mutex_enter(&ringp->s_ring_lock); 5035 cookie = 5036 mac_tx_sring_enqueue(ringp, mp_chain, flag, ret_mp); 5037 mutex_exit(&ringp->s_ring_lock); 5038 return (cookie); 5039 } 5040 SRS_TX_STATS_UPDATE(mac_srs, &stats); 5041 SOFTRING_TX_STATS_UPDATE(ringp, &stats); 5042 5043 return (0); 5044 } 5045 } 5046