1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License (the "License"). 6 * You may not use this file except in compliance with the License. 7 * 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9 * or http://www.opensolaris.org/os/licensing. 10 * See the License for the specific language governing permissions 11 * and limitations under the License. 12 * 13 * When distributing Covered Code, include this CDDL HEADER in each 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15 * If applicable, add the following below this CDDL HEADER, with the 16 * fields enclosed by brackets "[]" replaced with your own identifying 17 * information: Portions Copyright [yyyy] [name of copyright owner] 18 * 19 * CDDL HEADER END 20 */ 21 /* 22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved. 23 * Use is subject to license terms. 24 * Copyright 2017 Joyent, Inc. 25 * Copyright 2013 Nexenta Systems, Inc. All rights reserved. 26 */ 27 28 /* 29 * MAC data path 30 * 31 * The MAC data path is concerned with the flow of traffic from mac clients -- 32 * DLS, IP, etc. -- to various GLDv3 device drivers -- e1000g, vnic, aggr, 33 * ixgbe, etc. -- and from the GLDv3 device drivers back to clients. 34 * 35 * ----------- 36 * Terminology 37 * ----------- 38 * 39 * MAC uses a lot of different, but related terms that are associated with the 40 * design and structure of the data path. Before we cover other aspects, first 41 * let's review the terminology that MAC uses. 42 * 43 * MAC 44 * 45 * This driver. It interfaces with device drivers and provides abstractions 46 * that the rest of the system consumes. All data links -- things managed 47 * with dladm(1M), are accessed through MAC. 48 * 49 * GLDv3 DEVICE DRIVER 50 * 51 * A GLDv3 device driver refers to a driver, both for pseudo-devices and 52 * real devices, which implement the GLDv3 driver API. Common examples of 53 * these are igb and ixgbe, which are drivers for various Intel networking 54 * cards. These devices may or may not have various features, such as 55 * hardware rings and checksum offloading. For MAC, a GLDv3 device is the 56 * final point for the transmission of a packet and the starting point for 57 * the receipt of a packet. 58 * 59 * FLOWS 60 * 61 * At a high level, a flow refers to a series of packets that are related. 62 * Often times the term is used in the context of TCP to indicate a unique 63 * TCP connection and the traffic over it. However, a flow can exist at 64 * other levels of the system as well. MAC has a notion of a default flow 65 * which is used for all unicast traffic addressed to the address of a MAC 66 * device. For example, when a VNIC is created, a default flow is created 67 * for the VNIC's MAC address. In addition, flows are created for broadcast 68 * groups and a user may create a flow with flowadm(1M). 69 * 70 * CLASSIFICATION 71 * 72 * Classification refers to the notion of identifying an incoming frame 73 * based on its destination address and optionally its source addresses and 74 * doing different processing based on that information. Classification can 75 * be done in both hardware and software. In general, we usually only 76 * classify based on the layer two destination, eg. for Ethernet, the 77 * destination MAC address. 78 * 79 * The system also will do classification based on layer three and layer 80 * four properties. This is used to support things like flowadm(1M), which 81 * allows setting QoS and other properties on a per-flow basis. 82 * 83 * RING 84 * 85 * Conceptually, a ring represents a series of framed messages, often in a 86 * contiguous chunk of memory that acts as a circular buffer. Rings come in 87 * a couple of forms. Generally they are either a hardware construct (hw 88 * ring) or they are a software construct (sw ring) maintained by MAC. 89 * 90 * HW RING 91 * 92 * A hardware ring is a set of resources provided by a GLDv3 device driver 93 * (even if it is a pseudo-device). A hardware ring comes in two different 94 * forms: receive (rx) rings and transmit (tx) rings. An rx hw ring is 95 * something that has a unique DMA (direct memory access) region and 96 * generally supports some form of classification (though it isn't always 97 * used), as well as a means of generating an interrupt specific to that 98 * ring. For example, the device may generate a specific MSI-X for a PCI 99 * express device. A tx ring is similar, except that it is dedicated to 100 * transmission. It may also be a vector for enabling features such as VLAN 101 * tagging and large transmit offloading. It usually has its own dedicated 102 * interrupts for transmit being completed. 103 * 104 * SW RING 105 * 106 * A software ring is a construction of MAC. It represents the same thing 107 * that a hardware ring generally does, a collection of frames. However, 108 * instead of being in a contiguous ring of memory, they're instead linked 109 * by using the mblk_t's b_next pointer. Each frame may itself be multiple 110 * mblk_t's linked together by the b_cont pointer. A software ring always 111 * represents a collection of classified packets; however, it varies as to 112 * whether it uses only layer two information, or a combination of that and 113 * additional layer three and layer four data. 114 * 115 * FANOUT 116 * 117 * Fanout is the idea of spreading out the load of processing frames based 118 * on the source and destination information contained in the layer two, 119 * three, and four headers, such that the data can then be processed in 120 * parallel using multiple hardware threads. 121 * 122 * A fanout algorithm hashes the headers and uses that to place different 123 * flows into a bucket. The most important thing is that packets that are 124 * in the same flow end up in the same bucket. If they do not, performance 125 * can be adversely affected. Consider the case of TCP. TCP severely 126 * penalizes a connection if the data arrives out of order. If a given flow 127 * is processed on different CPUs, then the data will appear out of order, 128 * hence the invariant that fanout always hash a given flow to the same 129 * bucket and thus get processed on the same CPU. 130 * 131 * RECEIVE SIDE SCALING (RSS) 132 * 133 * 134 * Receive side scaling is a term that isn't common in illumos, but is used 135 * by vendors and was popularized by Microsoft. It refers to the idea of 136 * spreading the incoming receive load out across multiple interrupts which 137 * can be directed to different CPUs. This allows a device to leverage 138 * hardware rings even when it doesn't support hardware classification. The 139 * hardware uses an algorithm to perform fanout that ensures the flow 140 * invariant is maintained. 141 * 142 * SOFT RING SET 143 * 144 * A soft ring set, commonly abbreviated SRS, is a collection of rings and 145 * is used for both transmitting and receiving. It is maintained in the 146 * structure mac_soft_ring_set_t. A soft ring set is usually associated 147 * with flows, and coordinates both the use of hardware and software rings. 148 * Because the use of hardware rings can change as devices such as VNICs 149 * come and go, we always ensure that the set has software classification 150 * rules that correspond to the hardware classification rules from rings. 151 * 152 * Soft ring sets are also used for the enforcement of various QoS 153 * properties. For example, if a bandwidth limit has been placed on a 154 * specific flow or device, then that will be enforced by the soft ring 155 * set. 156 * 157 * SERVICE ATTACHMENT POINT (SAP) 158 * 159 * The service attachment point is a DLPI (Data Link Provider Interface) 160 * concept; however, it comes up quite often in MAC. Most MAC devices speak 161 * a protocol that has some notion of different channels or message type 162 * identifiers. For example, Ethernet defines an EtherType which is a part 163 * of the Ethernet header and defines the particular protocol of the data 164 * payload. If the EtherType is set to 0x0800, then it defines that the 165 * contents of that Ethernet frame is IPv4 traffic. For Ethernet, the 166 * EtherType is the SAP. 167 * 168 * In DLPI, a given consumer attaches to a specific SAP. In illumos, the ip 169 * and arp drivers attach to the EtherTypes for IPv4, IPv6, and ARP. Using 170 * libdlpi(3LIB) user software can attach to arbitrary SAPs. With the 171 * exception of 802.1Q VLAN tagged traffic, MAC itself does not directly 172 * consume the SAP; however, it uses that information as part of hashing 173 * and it may be used as part of the construction of flows. 174 * 175 * PRIMARY MAC CLIENT 176 * 177 * The primary mac client refers to a mac client whose unicast address 178 * matches the address of the device itself. For example, if the system has 179 * instance of the e1000g driver such as e1000g0, e1000g1, etc., the 180 * primary mac client is the one named after the device itself. VNICs that 181 * are created on top of such devices are not the primary client. 182 * 183 * TRANSMIT DESCRIPTORS 184 * 185 * Transmit descriptors are a resource that most GLDv3 device drivers have. 186 * Generally, a GLDv3 device driver takes a frame that's meant to be output 187 * and puts a copy of it into a region of memory. Each region of memory 188 * usually has an associated descriptor that the device uses to manage 189 * properties of the frames. Devices have a limited number of such 190 * descriptors. They get reclaimed once the device finishes putting the 191 * frame on the wire. 192 * 193 * If the driver runs out of transmit descriptors, for example, the OS is 194 * generating more frames than it can put on the wire, then it will return 195 * them back to the MAC layer. 196 * 197 * --------------------------------- 198 * Rings, Classification, and Fanout 199 * --------------------------------- 200 * 201 * The heart of MAC is made up of rings, and not those that Elven-kings wear. 202 * When receiving a packet, MAC breaks the work into two different, though 203 * interrelated phases. The first phase is generally classification and then the 204 * second phase is generally fanout. When a frame comes in from a GLDv3 Device, 205 * MAC needs to determine where that frame should be delivered. If it's a 206 * unicast frame (say a normal TCP/IP packet), then it will be delivered to a 207 * single MAC client; however, if it's a broadcast or multicast frame, then MAC 208 * may need to deliver it to multiple MAC clients. 209 * 210 * On transmit, classification isn't quite as important, but may still be used. 211 * Unlike with the receive path, the classification is not used to determine 212 * devices that should transmit something, but rather is used for special 213 * properties of a flow, eg. bandwidth limits for a given IP address, device, or 214 * connection. 215 * 216 * MAC employs a software classifier and leverages hardware classification as 217 * well. The software classifier can leverage the full layer two information, 218 * source, destination, VLAN, and SAP. If the SAP indicates that IP traffic is 219 * being sent, it can classify based on the IP header, and finally, it also 220 * knows how to classify based on the local and remote ports of TCP, UDP, and 221 * SCTP. 222 * 223 * Hardware classifiers vary in capability. Generally all hardware classifiers 224 * provide the capability to classify based on the destination MAC address. Some 225 * hardware has additional filters built in for performing more in-depth 226 * classification; however, it often has much more limited resources for these 227 * activities as compared to the layer two destination address classification. 228 * 229 * The modus operandi in MAC is to always ensure that we have software-based 230 * capabilities and rules in place and then to supplement that with hardware 231 * resources when available. In general, simple layer two classification is 232 * sufficient and nothing else is used, unless a specific flow is created with 233 * tools such as flowadm(1M) or bandwidth limits are set on a device with 234 * dladm(1M). 235 * 236 * RINGS AND GROUPS 237 * 238 * To get into how rings and classification play together, it's first important 239 * to understand how hardware devices commonly associate rings and allow them to 240 * be programmed. Recall that a hardware ring should be thought of as a DMA 241 * buffer and an interrupt resource. Rings are then collected into groups. A 242 * group itself has a series of classification rules. One or more MAC addresses 243 * are assigned to a group. 244 * 245 * Hardware devices vary in terms of what capabilities they provide. Sometimes 246 * they allow for a dynamic assignment of rings to a group and sometimes they 247 * have a static assignment of rings to a group. For example, the ixgbe driver 248 * has a static assignment of rings to groups such that every group has exactly 249 * one ring and the number of groups is equal to the number of rings. 250 * 251 * Classification and receive side scaling both come into play with how a device 252 * advertises itself to MAC and how MAC uses it. If a device supports layer two 253 * classification of frames, then MAC will assign MAC addresses to a group as a 254 * form of primary classification. If a single MAC address is assigned to a 255 * group, a common case, then MAC will consider packets that come in from rings 256 * on that group to be fully classified and will not need to do any software 257 * classification unless a specific flow has been created. 258 * 259 * If a device supports receive side scaling, then it may advertise or support 260 * groups with multiple rings. In those cases, then receive side scaling will 261 * come into play and MAC will use that as a means of fanning out received 262 * frames across multiple CPUs. This can also be combined with groups that 263 * support layer two classification. 264 * 265 * If a device supports dynamic assignments of rings to groups, then MAC will 266 * change around the way that rings are assigned to various groups as devices 267 * come and go from the system. For example, when a VNIC is created, a new flow 268 * will be created for the VNIC's MAC address. If a hardware ring is available, 269 * MAC may opt to reassign it from one group to another. 270 * 271 * ASSIGNMENT OF HARDWARE RINGS 272 * 273 * This is a bit of a complicated subject that varies depending on the device, 274 * the use of aggregations, the special nature of the primary mac client. This 275 * section deserves being fleshed out. 276 * 277 * FANOUT 278 * 279 * illumos uses fanout to help spread out the incoming processing load of chains 280 * of frames away from a single CPU. If a device supports receive side scaling, 281 * then that provides an initial form of fanout; however, what we're concerned 282 * with all happens after the context of a given set of frames being classified 283 * to a soft ring set. 284 * 285 * After frames reach a soft ring set and account for any potential bandwidth 286 * related accounting, they may be fanned out based on one of the following 287 * three modes: 288 * 289 * o No Fanout 290 * o Protocol level fanout 291 * o Full software ring protocol fanout 292 * 293 * MAC makes the determination as to which of these modes a given soft ring set 294 * obtains based on parameters such as whether or not it's the primary mac 295 * client, whether it's on a 10 GbE or faster device, user controlled dladm(1M) 296 * properties, and the nature of the hardware and the resources that it has. 297 * 298 * When there is no fanout, MAC does not create any soft rings for a device and 299 * the device has frames delivered directly to the MAC client. 300 * 301 * Otherwise, all fanout is performed by software. MAC divides incoming frames 302 * into one of three buckets -- IPv4 TCP traffic, IPv4 UDP traffic, and 303 * everything else. Note, VLAN tagged traffic is considered other, regardless of 304 * the interior EtherType. Regardless of the type of fanout, these three 305 * categories or buckets are always used. 306 * 307 * The difference between protocol level fanout and full software ring protocol 308 * fanout is the number of software rings that end up getting created. The 309 * system always uses the same number of software rings per protocol bucket. So 310 * in the first case when we're just doing protocol level fanout, we just create 311 * one software ring each for IPv4 TCP traffic, IPv4 UDP traffic, and everything 312 * else. 313 * 314 * In the case where we do full software ring protocol fanout, we generally use 315 * mac_compute_soft_ring_count() to determine the number of rings. There are 316 * other combinations of properties and devices that may send us down other 317 * paths, but this is a common starting point. If it's a non-bandwidth enforced 318 * device and we're on at least a 10 GbE link, then we'll use eight soft rings 319 * per protocol bucket as a starting point. See mac_compute_soft_ring_count() 320 * for more information on the total number. 321 * 322 * For each of these rings, we create a mac_soft_ring_t and an associated worker 323 * thread. Particularly when doing full software ring protocol fanout, we bind 324 * each of the worker threads to individual CPUs. 325 * 326 * The other advantage of these software rings is that it allows upper layers to 327 * optionally poll on them. For example, TCP can leverage an squeue to poll on 328 * the software ring, see squeue.c for more information. 329 * 330 * DLS BYPASS 331 * 332 * DLS is the data link services module. It interfaces with DLPI, which is the 333 * primary way that other parts of the system such as IP interface with the MAC 334 * layer. While DLS is traditionally a STREAMS-based interface, it allows for 335 * certain modules such as IP to negotiate various more modern interfaces to be 336 * used, which are useful for higher performance and allow it to use direct 337 * function calls to DLS instead of using STREAMS. 338 * 339 * When we have IPv4 TCP or UDP software rings, then traffic on those rings is 340 * eligible for what we call the dls bypass. In those cases, rather than going 341 * out mac_rx_deliver() to DLS, DLS instead registers them to go directly via 342 * the direct callback registered with DLS, generally ip_input(). 343 * 344 * HARDWARE RING POLLING 345 * 346 * GLDv3 devices with hardware rings generally deliver chains of messages 347 * (mblk_t chain) during the context of a single interrupt. However, interrupts 348 * are not the only way that these devices may be used. As part of implementing 349 * ring support, a GLDv3 device driver must have a way to disable the generation 350 * of that interrupt and allow for the operating system to poll on that ring. 351 * 352 * To implement this, every soft ring set has a worker thread and a polling 353 * thread. If a sufficient packet rate comes into the system, MAC will 'blank' 354 * (disable) interrupts on that specific ring and the polling thread will start 355 * consuming packets from the hardware device and deliver them to the soft ring 356 * set, where the worker thread will take over. 357 * 358 * Once the rate of packet intake drops down below a certain threshold, then 359 * polling on the hardware ring will be quiesced and interrupts will be 360 * re-enabled for the given ring. This effectively allows the system to shift 361 * how it handles a ring based on its load. At high packet rates, polling on the 362 * device as opposed to relying on interrupts can actually reduce overall system 363 * load due to the minimization of interrupt activity. 364 * 365 * Note the importance of each ring having its own interrupt source. The whole 366 * idea here is that we do not disable interrupts on the device as a whole, but 367 * rather each ring can be independently toggled. 368 * 369 * USE OF WORKER THREADS 370 * 371 * Both the soft ring set and individual soft rings have a worker thread 372 * associated with them that may be bound to a specific CPU in the system. Any 373 * such assignment will get reassessed as part of dynamic reconfiguration events 374 * in the system such as the onlining and offlining of CPUs and the creation of 375 * CPU partitions. 376 * 377 * In many cases, while in an interrupt, we try to deliver a frame all the way 378 * through the stack in the context of the interrupt itself. However, if the 379 * amount of queued frames has exceeded a threshold, then we instead defer to 380 * the worker thread to do this work and signal it. This is particularly useful 381 * when you have the soft ring set delivering frames into multiple software 382 * rings. If it was only delivering frames into a single software ring then 383 * there'd be no need to have another thread take over. However, if it's 384 * delivering chains of frames to multiple rings, then it's worthwhile to have 385 * the worker for the software ring take over so that the different software 386 * rings can be processed in parallel. 387 * 388 * In a similar fashion to the hardware polling thread, if we don't have a 389 * backlog or there's nothing to do, then the worker thread will go back to 390 * sleep and frames can be delivered all the way from an interrupt. This 391 * behavior is useful as it's designed to minimize latency and the default 392 * disposition of MAC is to optimize for latency. 393 * 394 * MAINTAINING CHAINS 395 * 396 * Another useful idea that MAC uses is to try and maintain frames in chains for 397 * as long as possible. The idea is that all of MAC can handle chains of frames 398 * structured as a series of mblk_t structures linked with the b_next pointer. 399 * When performing software classification and software fanout, MAC does not 400 * simply determine the destination and send the frame along. Instead, in the 401 * case of classification, it tries to maintain a chain for as long as possible 402 * before passing it along and performing additional processing. 403 * 404 * In the case of fanout, MAC first determines what the target software ring is 405 * for every frame in the original chain and constructs a new chain for each 406 * target. MAC then delivers the new chain to each software ring in succession. 407 * 408 * The whole rationale for doing this is that we want to try and maintain the 409 * pipe as much as possible and deliver as many frames through the stack at once 410 * that we can, rather than just pushing a single frame through. This can often 411 * help bring down latency and allows MAC to get a better sense of the overall 412 * activity in the system and properly engage worker threads. 413 * 414 * -------------------- 415 * Bandwidth Management 416 * -------------------- 417 * 418 * Bandwidth management is something that's built into the soft ring set itself. 419 * When bandwidth limits are placed on a flow, a corresponding soft ring set is 420 * toggled into bandwidth mode. This changes how we transmit and receive the 421 * frames in question. 422 * 423 * Bandwidth management is done on a per-tick basis. We translate the user's 424 * requested bandwidth from a quantity per-second into a quantity per-tick. MAC 425 * cannot process a frame across more than one tick, thus it sets a lower bound 426 * for the bandwidth cap to be a single MTU. This also means that when 427 * hires ticks are enabled (hz is set to 1000), that the minimum amount of 428 * bandwidth is higher, because the number of ticks has increased and MAC has to 429 * go from accepting 100 packets / sec to 1000 / sec. 430 * 431 * The bandwidth counter is reset by either the soft ring set's worker thread or 432 * a thread that is doing an inline transmit or receive if they discover that 433 * the current tick is in the future from the recorded tick. 434 * 435 * Whenever we're receiving or transmitting data, we end up leaving most of the 436 * work to the soft ring set's worker thread. This forces data inserted into the 437 * soft ring set to be effectively serialized and allows us to exhume bandwidth 438 * at a reasonable rate. If there is nothing in the soft ring set at the moment 439 * and the set has available bandwidth, then it may processed inline. 440 * Otherwise, the worker is responsible for taking care of the soft ring set. 441 * 442 * --------------------- 443 * The Receive Data Path 444 * --------------------- 445 * 446 * The following series of ASCII art images breaks apart the way that a frame 447 * comes in and is processed in MAC. 448 * 449 * Part 1 -- Initial frame receipt, SRS classification 450 * 451 * Here, a frame is received by a GLDv3 driver, generally in the context of an 452 * interrupt, and it ends up in mac_rx_common(). A driver calls either mac_rx or 453 * mac_rx_ring, depending on whether or not it supports rings and can identify 454 * the interrupt as having come from a specific ring. Here we determine whether 455 * or not it's fully classified and perform software classification as 456 * appropriate. From here, everything always ends up going to either entry [A] 457 * or entry [B] based on whether or not they have subflow processing needed. We 458 * leave via fanout or delivery. 459 * 460 * +===========+ 461 * v hardware v 462 * v interrupt v 463 * +===========+ 464 * | 465 * * . . appropriate 466 * | upcall made 467 * | by GLDv3 driver . . always 468 * | . 469 * +--------+ | +----------+ . +---------------+ 470 * | GLDv3 | +---->| mac_rx |-----*--->| mac_rx_common | 471 * | Driver |-->--+ +----------+ +---------------+ 472 * +--------+ | ^ | 473 * | | ^ v 474 * ^ | * . . always +----------------------+ 475 * | | | | mac_promisc_dispatch | 476 * | | +-------------+ +----------------------+ 477 * | +--->| mac_rx_ring | | 478 * | +-------------+ * . . hw classified 479 * | v or single flow? 480 * | | 481 * | +--------++--------------+ 482 * | | | * hw class, 483 * | | * hw classified | subflows 484 * | no hw class and . * | or single | exist 485 * | subflows | | flow | 486 * | | v v 487 * | | +-----------+ +-----------+ 488 * | | | goto | | goto | 489 * | | | entry [A] | | entry [B] | 490 * | | +-----------+ +-----------+ 491 * | v ^ 492 * | +-------------+ | 493 * | | mac_rx_flow | * SRS and flow found, 494 * | +-------------+ | call flow cb 495 * | | +------+ 496 * | v | 497 * v +==========+ +-----------------+ 498 * | v For each v--->| mac_rx_classify | 499 * +----------+ v mblk_t v +-----------------+ 500 * | srs | +==========+ 501 * | pollling | 502 * | thread |->------------------------------------------+ 503 * +----------+ | 504 * v . inline 505 * +--------------------+ +----------+ +---------+ . 506 * [A]---->| mac_rx_srs_process |-->| check bw |-->| enqueue |--*---------+ 507 * +--------------------+ | limits | | frames | | 508 * ^ +----------+ | to SRS | | 509 * | +---------+ | 510 * | send chain +--------+ | | 511 * * when clasified | signal | * BW limits, | 512 * | flow changes | srs |<---+ loopback, | 513 * | | worker | stack too | 514 * | +--------+ deep | 515 * +-----------------+ +--------+ | 516 * | mac_flow_lookup | | srs | +---------------------+ | 517 * +-----------------+ | worker |---->| mac_rx_srs_drain |<---+ 518 * ^ | thread | | mac_rx_srs_drain_bw | 519 * | +--------+ +---------------------+ 520 * | | 521 * +----------------------------+ * software rings 522 * [B]-->| mac_rx_srs_subflow_process | | for fanout? 523 * +----------------------------+ | 524 * +----------+-----------+ 525 * | | 526 * v v 527 * +--------+ +--------+ 528 * | goto | | goto | 529 * | Part 2 | | Part 3 | 530 * +--------+ +--------+ 531 * 532 * Part 2 -- Fanout 533 * 534 * This part is concerned with using software fanout to assign frames to 535 * software rings and then deliver them to MAC clients or allow those rings to 536 * be polled upon. While there are two different primary fanout entry points, 537 * mac_rx_fanout and mac_rx_proto_fanout, they behave in similar ways, and aside 538 * from some of the individual hashing techniques used, most of the general 539 * flow is the same. 540 * 541 * +--------+ +-------------------+ 542 * | From |---+--------->| mac_rx_srs_fanout |----+ 543 * | Part 1 | | +-------------------+ | +=================+ 544 * +--------+ | | v for each mblk_t v 545 * * . . protocol only +--->v assign to new v 546 * | fanout | v chain based on v 547 * | | v hash % nrings v 548 * | +-------------------------+ | +=================+ 549 * +--->| mac_rx_srs_proto_fanout |----+ | 550 * +-------------------------+ | 551 * v 552 * +------------+ +--------------------------+ +================+ 553 * | enqueue in |<---| mac_rx_soft_ring_process |<------v for each chain v 554 * | soft ring | +--------------------------+ +================+ 555 * +------------+ 556 * | +-----------+ 557 * * soft ring set | soft ring | 558 * | empty and no | worker | 559 * | worker? | thread | 560 * | +-----------+ 561 * +------*----------------+ | 562 * | . | v 563 * No . * . Yes | +------------------------+ 564 * | +----<--| mac_rx_soft_ring_drain | 565 * | | +------------------------+ 566 * v | 567 * +-----------+ v 568 * | signal | +---------------+ 569 * | soft ring | | Deliver chain | 570 * | worker | | goto Part 3 | 571 * +-----------+ +---------------+ 572 * 573 * 574 * Part 3 -- Packet Delivery 575 * 576 * Here, we go through and deliver the mblk_t chain directly to a given 577 * processing function. In a lot of cases this is mac_rx_deliver(). In the case 578 * of DLS bypass being used, then instead we end up going ahead and deliver it 579 * to the direct callback registered with DLS, generally ip_input. 580 * 581 * 582 * +---------+ +----------------+ +------------------+ 583 * | From |---+------->| mac_rx_deliver |--->| Off to DLS, or | 584 * | Parts 1 | | +----------------+ | other MAC client | 585 * | and 2 | * DLS bypass +------------------+ 586 * +---------+ | enabled +----------+ +-------------+ 587 * +---------->| ip_input |--->| To IP | 588 * +----------+ | and beyond! | 589 * +-------------+ 590 * 591 * ---------------------- 592 * The Transmit Data Path 593 * ---------------------- 594 * 595 * Before we go into the images, it's worth talking about a problem that is a 596 * bit different from the receive data path. GLDv3 device drivers have a finite 597 * amount of transmit descriptors. When they run out, they return unused frames 598 * back to MAC. MAC, at this point has several options about what it will do, 599 * which vary based upon the settings that the client uses. 600 * 601 * When a device runs out of descriptors, the next thing that MAC does is 602 * enqueue them off of the soft ring set or a software ring, depending on the 603 * configuration of the soft ring set. MAC will enqueue up to a high watermark 604 * of mblk_t chains, at which point it will indicate flow control back to the 605 * client. Once this condition is reached, any mblk_t chains that were not 606 * enqueued will be returned to the caller and they will have to decide what to 607 * do with them. There are various flags that control this behavior that a 608 * client may pass, which are discussed below. 609 * 610 * When this condition is hit, MAC also returns a cookie to the client in 611 * addition to unconsumed frames. Clients can poll on that cookie and register a 612 * callback with MAC to be notified when they are no longer subject to flow 613 * control, at which point they may continue to call mac_tx(). This flow control 614 * actually manages to work itself all the way up the stack, back through dls, 615 * to ip, through the various protocols, and to sockfs. 616 * 617 * While the behavior described above is the default, this behavior can be 618 * modified. There are two alternate modes, described below, which are 619 * controlled with flags. 620 * 621 * DROP MODE 622 * 623 * This mode is controlled by having the client pass the MAC_DROP_ON_NO_DESC 624 * flag. When this is passed, if a device driver runs out of transmit 625 * descriptors, then the MAC layer will drop any unsent traffic. The client in 626 * this case will never have any frames returned to it. 627 * 628 * DON'T ENQUEUE 629 * 630 * This mode is controlled by having the client pass the MAC_TX_NO_ENQUEUE flag. 631 * If the MAC_DROP_ON_NO_DESC flag is also passed, it takes precedence. In this 632 * mode, when we hit a case where a driver runs out of transmit descriptors, 633 * then instead of enqueuing packets in a soft ring set or software ring, we 634 * instead return the mblk_t chain back to the caller and immediately put the 635 * soft ring set into flow control mode. 636 * 637 * The following series of ASCII art images describe the transmit data path that 638 * MAC clients enter into based on calling into mac_tx(). A soft ring set has a 639 * transmission function associated with it. There are seven possible 640 * transmission modes, some of which share function entry points. The one that a 641 * soft ring set gets depends on properties such as whether there are 642 * transmission rings for fanout, whether the device involves aggregations, 643 * whether any bandwidth limits exist, etc. 644 * 645 * 646 * Part 1 -- Initial checks 647 * 648 * * . called by 649 * | MAC clients 650 * v . . No 651 * +--------+ +-----------+ . +-------------------+ +====================+ 652 * | mac_tx |->| device |-*-->| mac_protect_check |->v Is this the simple v 653 * +--------+ | quiesced? | +-------------------+ v case? See [1] v 654 * +-----------+ | +====================+ 655 * * . Yes * failed | 656 * v | frames | 657 * +--------------+ | +-------+---------+ 658 * | freemsgchain |<---------+ Yes . * No . * 659 * +--------------+ v v 660 * +-----------+ +--------+ 661 * | goto | | goto | 662 * | Part 2 | | SRS TX | 663 * | Entry [A] | | func | 664 * +-----------+ +--------+ 665 * | | 666 * | v 667 * | +--------+ 668 * +---------->| return | 669 * | cookie | 670 * +--------+ 671 * 672 * [1] The simple case refers to the SRS being configured with the 673 * SRS_TX_DEFAULT transmission mode, having a single mblk_t (not a chain), their 674 * being only a single active client, and not having a backlog in the srs. 675 * 676 * 677 * Part 2 -- The SRS transmission functions 678 * 679 * This part is a bit more complicated. The different transmission paths often 680 * leverage one another. In this case, we'll draw out the more common ones 681 * before the parts that depend upon them. Here, we're going to start with the 682 * workings of mac_tx_send() a common function that most of the others end up 683 * calling. 684 * 685 * +-------------+ 686 * | mac_tx_send | 687 * +-------------+ 688 * | 689 * v 690 * +=============+ +==============+ 691 * v more than v--->v check v 692 * v one client? v v VLAN and add v 693 * +=============+ v VLAN tags v 694 * | +==============+ 695 * | | 696 * +------------------+ 697 * | 698 * | [A] 699 * v | 700 * +============+ . No v 701 * v more than v . +==========+ +--------------------------+ 702 * v one active v-*---->v for each v---->| mac_promisc_dispatch_one |---+ 703 * v client? v v mblk_t v +--------------------------+ | 704 * +============+ +==========+ ^ | 705 * | | +==========+ | 706 * * . Yes | v hardware v<-------+ 707 * v +------------+ v rings? v 708 * +==========+ | +==========+ 709 * v for each v No . . . * | 710 * v mblk_t v specific | | 711 * +==========+ flow | +-----+-----+ 712 * | | | | 713 * v | v v 714 * +-----------------+ | +-------+ +---------+ 715 * | mac_tx_classify |------------+ | GLDv3 | | GLDv3 | 716 * +-----------------+ |TX func| | ring tx | 717 * | +-------+ | func | 718 * * Specific flow, generally | +---------+ 719 * | bcast, mcast, loopback | | 720 * v +-----+-----+ 721 * +==========+ +---------+ | 722 * v valid L2 v--*--->| freemsg | v 723 * v header v . No +---------+ +-------------------+ 724 * +==========+ | return unconsumed | 725 * * . Yes | frames to the | 726 * v | caller | 727 * +===========+ +-------------------+ 728 * v braodcast v +----------------+ ^ 729 * v flow? v--*-->| mac_bcast_send |------------------+ 730 * +===========+ . +----------------+ | 731 * | . . Yes | 732 * No . * v 733 * | +---------------------+ +---------------+ +----------+ 734 * +->|mac_promisc_dispatch |->| mac_fix_cksum |->| flow | 735 * +---------------------+ +---------------+ | callback | 736 * +----------+ 737 * 738 * 739 * In addition, many but not all of the routines, all rely on 740 * mac_tx_softring_process as an entry point. 741 * 742 * 743 * . No . No 744 * +--------------------------+ +========+ . +===========+ . +-------------+ 745 * | mac_tx_soft_ring_process |-->v worker v-*->v out of tx v-*->| goto | 746 * +--------------------------+ v only? v v descr.? v | mac_tx_send | 747 * +========+ +===========+ +-------------+ 748 * Yes . * * . Yes | 749 * . No v | v 750 * v=========+ . +===========+ . Yes | Yes . +==========+ 751 * v apppend v<--*----------v out of tx v-*-------+---------*--v returned v 752 * v mblk_t v v descr.? v | v frames? v 753 * v chain v +===========+ | +==========+ 754 * +=========+ | *. No 755 * | | v 756 * v v +------------+ 757 * +===================+ +----------------------+ | done | 758 * v worker scheduled? v | mac_tx_sring_enqueue | | processing | 759 * v Out of tx descr? v +----------------------+ +------------+ 760 * +===================+ | 761 * | | . Yes v 762 * * Yes * No . +============+ 763 * | v +-*---------v drop on no v 764 * | +========+ v v TX desc? v 765 * | v wake v +----------+ +============+ 766 * | v worker v | mac_pkt_ | * . No 767 * | +========+ | drop | | . Yes . No 768 * | | +----------+ v . . 769 * | | v ^ +===============+ . +========+ . 770 * +--+--------+---------+ | v Don't enqueue v-*->v ring v-*----+ 771 * | | v Set? v v empty? v | 772 * | +---------------+ +===============+ +========+ | 773 * | | | | | 774 * | | +-------------------+ | | 775 * | *. Yes | +---------+ | 776 * | | v v v 777 * | | +===========+ +========+ +--------------+ 778 * | +<-v At hiwat? v v append v | return | 779 * | +===========+ v mblk_t v | mblk_t chain | 780 * | * No v chain v | and flow | 781 * | v +========+ | control | 782 * | +=========+ | | cookie | 783 * | v append v v +--------------+ 784 * | v mblk_t v +========+ 785 * | v chain v v wake v +------------+ 786 * | +=========+ v worker v-->| done | 787 * | | +========+ | processing | 788 * | v .. Yes +------------+ 789 * | +=========+ . +========+ 790 * | v first v--*-->v wake v 791 * | v append? v v worker v 792 * | +=========+ +========+ 793 * | | | 794 * | No . * | 795 * | v | 796 * | +--------------+ | 797 * +------>| Return | | 798 * | flow control |<------------+ 799 * | cookie | 800 * +--------------+ 801 * 802 * 803 * The remaining images are all specific to each of the different transmission 804 * modes. 805 * 806 * SRS TX DEFAULT 807 * 808 * [ From Part 1 ] 809 * | 810 * v 811 * +-------------------------+ 812 * | mac_tx_single_ring_mode | 813 * +-------------------------+ 814 * | 815 * | . Yes 816 * v . 817 * +==========+ . +============+ 818 * v SRS v-*->v Try to v---->---------------------+ 819 * v backlog? v v enqueue in v | 820 * +==========+ v SRS v-->------+ * . . Queue too 821 * | +============+ * don't enqueue | deep or 822 * * . No ^ | | flag or at | drop flag 823 * | | v | hiwat, | 824 * v | | | return +---------+ 825 * +-------------+ | | | cookie | freemsg | 826 * | goto |-*-----+ | | +---------+ 827 * | mac_tx_send | . returned | | | 828 * +-------------+ mblk_t | | | 829 * | | | | 830 * | | | | 831 * * . . all mblk_t * queued, | | 832 * v consumed | may return | | 833 * +-------------+ | tx cookie | | 834 * | SRS TX func |<------------+------------+----------------+ 835 * | completed | 836 * +-------------+ 837 * 838 * SRS_TX_SERIALIZE 839 * 840 * +------------------------+ 841 * | mac_tx_serializer_mode | 842 * +------------------------+ 843 * | 844 * | . No 845 * v . 846 * +============+ . +============+ +-------------+ +============+ 847 * v srs being v-*->v set SRS v--->| goto |-->v remove SRS v 848 * v processed? v v proc flags v | mac_tx_send | v proc flag v 849 * +============+ +============+ +-------------+ +============+ 850 * | | 851 * * Yes | 852 * v . No v 853 * +--------------------+ . +==========+ 854 * | mac_tx_srs_enqueue | +------------------------*-----<--v returned v 855 * +--------------------+ | v frames? v 856 * | | . Yes +==========+ 857 * | | . | 858 * | | . +=========+ v 859 * v +-<-*-v queued v +--------------------+ 860 * +-------------+ | v frames? v<----| mac_tx_srs_enqueue | 861 * | SRS TX func | | +=========+ +--------------------+ 862 * | completed, |<------+ * . Yes 863 * | may return | | v 864 * | cookie | | +========+ 865 * +-------------+ +-<---v wake v 866 * v worker v 867 * +========+ 868 * 869 * 870 * SRS_TX_FANOUT 871 * 872 * . Yes 873 * +--------------------+ +=============+ . +--------------------------+ 874 * | mac_tx_fanout_mode |--->v Have fanout v-*-->| goto | 875 * +--------------------+ v hint? v | mac_rx_soft_ring_process | 876 * +=============+ +--------------------------+ 877 * * . No | 878 * v ^ 879 * +===========+ | 880 * +--->v for each v +===============+ 881 * | v mblk_t v v pick softring v 882 * same * +===========+ v from hash v 883 * hash | | +===============+ 884 * | v | 885 * | +--------------+ | 886 * +---| mac_pkt_hash |--->*------------+ 887 * +--------------+ . different 888 * hash or 889 * done proc. 890 * SRS_TX_AGGR chain 891 * 892 * +------------------+ +================================+ 893 * | mac_tx_aggr_mode |--->v Use aggr capab function to v 894 * +------------------+ v find appropriate tx ring. v 895 * v Applies hash based on aggr v 896 * v policy, see mac_tx_aggr_mode() v 897 * +================================+ 898 * | 899 * v 900 * +-------------------------------+ 901 * | goto | 902 * | mac_rx_srs_soft_ring_process | 903 * +-------------------------------+ 904 * 905 * 906 * SRS_TX_BW, SRS_TX_BW_FANOUT, SRS_TX_BW_AGGR 907 * 908 * Note, all three of these tx functions start from the same place -- 909 * mac_tx_bw_mode(). 910 * 911 * +----------------+ 912 * | mac_tx_bw_mode | 913 * +----------------+ 914 * | 915 * v . No . No . Yes 916 * +==============+ . +============+ . +=============+ . +=========+ 917 * v Out of BW? v--*->v SRS empty? v--*->v reset BW v-*->v Bump BW v 918 * +==============+ +============+ v tick count? v v Usage v 919 * | | +=============+ +=========+ 920 * | +---------+ | | 921 * | | +--------------------+ | 922 * | | | +----------------------+ 923 * v | v v 924 * +===============+ | +==========+ +==========+ +------------------+ 925 * v Don't enqueue v | v set bw v v Is aggr? v--*-->| goto | 926 * v flag set? v | v enforced v +==========+ . | mac_tx_aggr_mode |-+ 927 * +===============+ | +==========+ | . +------------------+ | 928 * | Yes .* | | No . * . | 929 * | | | | | . Yes | 930 * * . No | | v | | 931 * | +---------+ | +========+ v +======+ | 932 * | | freemsg | | v append v +============+ . Yes v pick v | 933 * | +---------+ | v mblk_t v v Is fanout? v--*---->v ring v | 934 * | | | v chain v +============+ +======+ | 935 * +------+ | +========+ | | | 936 * v | | v v | 937 * +---------+ | v +-------------+ +--------------------+ | 938 * | return | | +========+ | goto | | goto | | 939 * | flow | | v wakeup v | mac_tx_send | | mac_tx_fanout_mode | | 940 * | control | | v worker v +-------------+ +--------------------+ | 941 * | cookie | | +========+ | | | 942 * +---------+ | | | +------+------+ 943 * | v | | 944 * | +---------+ | v 945 * | | return | +============+ +------------+ 946 * | | flow | v unconsumed v-------+ | done | 947 * | | control | v frames? v | | processing | 948 * | | cookie | +============+ | +------------+ 949 * | +---------+ | | 950 * | Yes * | 951 * | | | 952 * | +===========+ | 953 * | v subtract v | 954 * | v unused bw v | 955 * | +===========+ | 956 * | | | 957 * | v | 958 * | +--------------------+ | 959 * +------------->| mac_tx_srs_enqueue | | 960 * +--------------------+ | 961 * | | 962 * | | 963 * +------------+ | 964 * | return fc | | 965 * | cookie and |<------+ 966 * | mblk_t | 967 * +------------+ 968 */ 969 970 #include <sys/types.h> 971 #include <sys/callb.h> 972 #include <sys/sdt.h> 973 #include <sys/strsubr.h> 974 #include <sys/strsun.h> 975 #include <sys/vlan.h> 976 #include <sys/stack.h> 977 #include <sys/archsystm.h> 978 #include <inet/ipsec_impl.h> 979 #include <inet/ip_impl.h> 980 #include <inet/sadb.h> 981 #include <inet/ipsecesp.h> 982 #include <inet/ipsecah.h> 983 #include <inet/ip6.h> 984 985 #include <sys/mac_impl.h> 986 #include <sys/mac_client_impl.h> 987 #include <sys/mac_client_priv.h> 988 #include <sys/mac_soft_ring.h> 989 #include <sys/mac_flow_impl.h> 990 991 static mac_tx_cookie_t mac_tx_single_ring_mode(mac_soft_ring_set_t *, mblk_t *, 992 uintptr_t, uint16_t, mblk_t **); 993 static mac_tx_cookie_t mac_tx_serializer_mode(mac_soft_ring_set_t *, mblk_t *, 994 uintptr_t, uint16_t, mblk_t **); 995 static mac_tx_cookie_t mac_tx_fanout_mode(mac_soft_ring_set_t *, mblk_t *, 996 uintptr_t, uint16_t, mblk_t **); 997 static mac_tx_cookie_t mac_tx_bw_mode(mac_soft_ring_set_t *, mblk_t *, 998 uintptr_t, uint16_t, mblk_t **); 999 static mac_tx_cookie_t mac_tx_aggr_mode(mac_soft_ring_set_t *, mblk_t *, 1000 uintptr_t, uint16_t, mblk_t **); 1001 1002 typedef struct mac_tx_mode_s { 1003 mac_tx_srs_mode_t mac_tx_mode; 1004 mac_tx_func_t mac_tx_func; 1005 } mac_tx_mode_t; 1006 1007 /* 1008 * There are seven modes of operation on the Tx side. These modes get set 1009 * in mac_tx_srs_setup(). Except for the experimental TX_SERIALIZE mode, 1010 * none of the other modes are user configurable. They get selected by 1011 * the system depending upon whether the link (or flow) has multiple Tx 1012 * rings or a bandwidth configured, or if the link is an aggr, etc. 1013 * 1014 * When the Tx SRS is operating in aggr mode (st_mode) or if there are 1015 * multiple Tx rings owned by Tx SRS, then each Tx ring (pseudo or 1016 * otherwise) will have a soft ring associated with it. These soft rings 1017 * are stored in srs_tx_soft_rings[] array. 1018 * 1019 * Additionally in the case of aggr, there is the st_soft_rings[] array 1020 * in the mac_srs_tx_t structure. This array is used to store the same 1021 * set of soft rings that are present in srs_tx_soft_rings[] array but 1022 * in a different manner. The soft ring associated with the pseudo Tx 1023 * ring is saved at mr_index (of the pseudo ring) in st_soft_rings[] 1024 * array. This helps in quickly getting the soft ring associated with the 1025 * Tx ring when aggr_find_tx_ring() returns the pseudo Tx ring that is to 1026 * be used for transmit. 1027 */ 1028 mac_tx_mode_t mac_tx_mode_list[] = { 1029 {SRS_TX_DEFAULT, mac_tx_single_ring_mode}, 1030 {SRS_TX_SERIALIZE, mac_tx_serializer_mode}, 1031 {SRS_TX_FANOUT, mac_tx_fanout_mode}, 1032 {SRS_TX_BW, mac_tx_bw_mode}, 1033 {SRS_TX_BW_FANOUT, mac_tx_bw_mode}, 1034 {SRS_TX_AGGR, mac_tx_aggr_mode}, 1035 {SRS_TX_BW_AGGR, mac_tx_bw_mode} 1036 }; 1037 1038 /* 1039 * Soft Ring Set (SRS) - The Run time code that deals with 1040 * dynamic polling from the hardware, bandwidth enforcement, 1041 * fanout etc. 1042 * 1043 * We try to use H/W classification on NIC and assign traffic for 1044 * a MAC address to a particular Rx ring or ring group. There is a 1045 * 1-1 mapping between a SRS and a Rx ring. The SRS dynamically 1046 * switches the underlying Rx ring between interrupt and 1047 * polling mode and enforces any specified B/W control. 1048 * 1049 * There is always a SRS created and tied to each H/W and S/W rule. 1050 * Whenever we create a H/W rule, we always add the the same rule to 1051 * S/W classifier and tie a SRS to it. 1052 * 1053 * In case a B/W control is specified, it is broken into bytes 1054 * per ticks and as soon as the quota for a tick is exhausted, 1055 * the underlying Rx ring is forced into poll mode for remainder of 1056 * the tick. The SRS poll thread only polls for bytes that are 1057 * allowed to come in the SRS. We typically let 4x the configured 1058 * B/W worth of packets to come in the SRS (to prevent unnecessary 1059 * drops due to bursts) but only process the specified amount. 1060 * 1061 * A MAC client (e.g. a VNIC or aggr) can have 1 or more 1062 * Rx rings (and corresponding SRSs) assigned to it. The SRS 1063 * in turn can have softrings to do protocol level fanout or 1064 * softrings to do S/W based fanout or both. In case the NIC 1065 * has no Rx rings, we do S/W classification to respective SRS. 1066 * The S/W classification rule is always setup and ready. This 1067 * allows the MAC layer to reassign Rx rings whenever needed 1068 * but packets still continue to flow via the default path and 1069 * getting S/W classified to correct SRS. 1070 * 1071 * The SRS's are used on both Tx and Rx side. They use the same 1072 * data structure but the processing routines have slightly different 1073 * semantics due to the fact that Rx side needs to do dynamic 1074 * polling etc. 1075 * 1076 * Dynamic Polling Notes 1077 * ===================== 1078 * 1079 * Each Soft ring set is capable of switching its Rx ring between 1080 * interrupt and poll mode and actively 'polls' for packets in 1081 * poll mode. If the SRS is implementing a B/W limit, it makes 1082 * sure that only Max allowed packets are pulled in poll mode 1083 * and goes to poll mode as soon as B/W limit is exceeded. As 1084 * such, there are no overheads to implement B/W limits. 1085 * 1086 * In poll mode, its better to keep the pipeline going where the 1087 * SRS worker thread keeps processing packets and poll thread 1088 * keeps bringing more packets (specially if they get to run 1089 * on different CPUs). This also prevents the overheads associated 1090 * by excessive signalling (on NUMA machines, this can be 1091 * pretty devastating). The exception is latency optimized case 1092 * where worker thread does no work and interrupt and poll thread 1093 * are allowed to do their own drain. 1094 * 1095 * We use the following policy to control Dynamic Polling: 1096 * 1) We switch to poll mode anytime the processing 1097 * thread causes a backlog to build up in SRS and 1098 * its associated Soft Rings (sr_poll_pkt_cnt > 0). 1099 * 2) As long as the backlog stays under the low water 1100 * mark (sr_lowat), we poll the H/W for more packets. 1101 * 3) If the backlog (sr_poll_pkt_cnt) exceeds low 1102 * water mark, we stay in poll mode but don't poll 1103 * the H/W for more packets. 1104 * 4) Anytime in polling mode, if we poll the H/W for 1105 * packets and find nothing plus we have an existing 1106 * backlog (sr_poll_pkt_cnt > 0), we stay in polling 1107 * mode but don't poll the H/W for packets anymore 1108 * (let the polling thread go to sleep). 1109 * 5) Once the backlog is relived (packets are processed) 1110 * we reenable polling (by signalling the poll thread) 1111 * only when the backlog dips below sr_poll_thres. 1112 * 6) sr_hiwat is used exclusively when we are not 1113 * polling capable and is used to decide when to 1114 * drop packets so the SRS queue length doesn't grow 1115 * infinitely. 1116 * 1117 * NOTE: Also see the block level comment on top of mac_soft_ring.c 1118 */ 1119 1120 /* 1121 * mac_latency_optimize 1122 * 1123 * Controls whether the poll thread can process the packets inline 1124 * or let the SRS worker thread do the processing. This applies if 1125 * the SRS was not being processed. For latency sensitive traffic, 1126 * this needs to be true to allow inline processing. For throughput 1127 * under load, this should be false. 1128 * 1129 * This (and other similar) tunable should be rolled into a link 1130 * or flow specific workload hint that can be set using dladm 1131 * linkprop (instead of multiple such tunables). 1132 */ 1133 boolean_t mac_latency_optimize = B_TRUE; 1134 1135 /* 1136 * MAC_RX_SRS_ENQUEUE_CHAIN and MAC_TX_SRS_ENQUEUE_CHAIN 1137 * 1138 * queue a mp or chain in soft ring set and increment the 1139 * local count (srs_count) for the SRS and the shared counter 1140 * (srs_poll_pkt_cnt - shared between SRS and its soft rings 1141 * to track the total unprocessed packets for polling to work 1142 * correctly). 1143 * 1144 * The size (total bytes queued) counters are incremented only 1145 * if we are doing B/W control. 1146 */ 1147 #define MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1148 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1149 if ((mac_srs)->srs_last != NULL) \ 1150 (mac_srs)->srs_last->b_next = (head); \ 1151 else \ 1152 (mac_srs)->srs_first = (head); \ 1153 (mac_srs)->srs_last = (tail); \ 1154 (mac_srs)->srs_count += count; \ 1155 } 1156 1157 #define MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1158 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \ 1159 \ 1160 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \ 1161 srs_rx->sr_poll_pkt_cnt += count; \ 1162 ASSERT(srs_rx->sr_poll_pkt_cnt > 0); \ 1163 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \ 1164 (mac_srs)->srs_size += (sz); \ 1165 mutex_enter(&(mac_srs)->srs_bw->mac_bw_lock); \ 1166 (mac_srs)->srs_bw->mac_bw_sz += (sz); \ 1167 mutex_exit(&(mac_srs)->srs_bw->mac_bw_lock); \ 1168 } \ 1169 } 1170 1171 #define MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \ 1172 mac_srs->srs_state |= SRS_ENQUEUED; \ 1173 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \ 1174 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \ 1175 (mac_srs)->srs_size += (sz); \ 1176 (mac_srs)->srs_bw->mac_bw_sz += (sz); \ 1177 } \ 1178 } 1179 1180 /* 1181 * Turn polling on routines 1182 */ 1183 #define MAC_SRS_POLLING_ON(mac_srs) { \ 1184 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1185 if (((mac_srs)->srs_state & \ 1186 (SRS_POLLING_CAPAB|SRS_POLLING)) == SRS_POLLING_CAPAB) { \ 1187 (mac_srs)->srs_state |= SRS_POLLING; \ 1188 (void) mac_hwring_disable_intr((mac_ring_handle_t) \ 1189 (mac_srs)->srs_ring); \ 1190 (mac_srs)->srs_rx.sr_poll_on++; \ 1191 } \ 1192 } 1193 1194 #define MAC_SRS_WORKER_POLLING_ON(mac_srs) { \ 1195 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1196 if (((mac_srs)->srs_state & \ 1197 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_POLLING)) == \ 1198 (SRS_POLLING_CAPAB|SRS_WORKER)) { \ 1199 (mac_srs)->srs_state |= SRS_POLLING; \ 1200 (void) mac_hwring_disable_intr((mac_ring_handle_t) \ 1201 (mac_srs)->srs_ring); \ 1202 (mac_srs)->srs_rx.sr_worker_poll_on++; \ 1203 } \ 1204 } 1205 1206 /* 1207 * MAC_SRS_POLL_RING 1208 * 1209 * Signal the SRS poll thread to poll the underlying H/W ring 1210 * provided it wasn't already polling (SRS_GET_PKTS was set). 1211 * 1212 * Poll thread gets to run only from mac_rx_srs_drain() and only 1213 * if the drain was being done by the worker thread. 1214 */ 1215 #define MAC_SRS_POLL_RING(mac_srs) { \ 1216 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \ 1217 \ 1218 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1219 srs_rx->sr_poll_thr_sig++; \ 1220 if (((mac_srs)->srs_state & \ 1221 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_GET_PKTS)) == \ 1222 (SRS_WORKER|SRS_POLLING_CAPAB)) { \ 1223 (mac_srs)->srs_state |= SRS_GET_PKTS; \ 1224 cv_signal(&(mac_srs)->srs_cv); \ 1225 } else { \ 1226 srs_rx->sr_poll_thr_busy++; \ 1227 } \ 1228 } 1229 1230 /* 1231 * MAC_SRS_CHECK_BW_CONTROL 1232 * 1233 * Check to see if next tick has started so we can reset the 1234 * SRS_BW_ENFORCED flag and allow more packets to come in the 1235 * system. 1236 */ 1237 #define MAC_SRS_CHECK_BW_CONTROL(mac_srs) { \ 1238 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1239 ASSERT(((mac_srs)->srs_type & SRST_TX) || \ 1240 MUTEX_HELD(&(mac_srs)->srs_bw->mac_bw_lock)); \ 1241 clock_t now = ddi_get_lbolt(); \ 1242 if ((mac_srs)->srs_bw->mac_bw_curr_time != now) { \ 1243 (mac_srs)->srs_bw->mac_bw_curr_time = now; \ 1244 (mac_srs)->srs_bw->mac_bw_used = 0; \ 1245 if ((mac_srs)->srs_bw->mac_bw_state & SRS_BW_ENFORCED) \ 1246 (mac_srs)->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; \ 1247 } \ 1248 } 1249 1250 /* 1251 * MAC_SRS_WORKER_WAKEUP 1252 * 1253 * Wake up the SRS worker thread to process the queue as long as 1254 * no one else is processing the queue. If we are optimizing for 1255 * latency, we wake up the worker thread immediately or else we 1256 * wait mac_srs_worker_wakeup_ticks before worker thread gets 1257 * woken up. 1258 */ 1259 int mac_srs_worker_wakeup_ticks = 0; 1260 #define MAC_SRS_WORKER_WAKEUP(mac_srs) { \ 1261 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \ 1262 if (!((mac_srs)->srs_state & SRS_PROC) && \ 1263 (mac_srs)->srs_tid == NULL) { \ 1264 if (((mac_srs)->srs_state & SRS_LATENCY_OPT) || \ 1265 (mac_srs_worker_wakeup_ticks == 0)) \ 1266 cv_signal(&(mac_srs)->srs_async); \ 1267 else \ 1268 (mac_srs)->srs_tid = \ 1269 timeout(mac_srs_fire, (mac_srs), \ 1270 mac_srs_worker_wakeup_ticks); \ 1271 } \ 1272 } 1273 1274 #define TX_BANDWIDTH_MODE(mac_srs) \ 1275 ((mac_srs)->srs_tx.st_mode == SRS_TX_BW || \ 1276 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_FANOUT || \ 1277 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_AGGR) 1278 1279 #define TX_SRS_TO_SOFT_RING(mac_srs, head, hint) { \ 1280 if (tx_mode == SRS_TX_BW_FANOUT) \ 1281 (void) mac_tx_fanout_mode(mac_srs, head, hint, 0, NULL);\ 1282 else \ 1283 (void) mac_tx_aggr_mode(mac_srs, head, hint, 0, NULL); \ 1284 } 1285 1286 /* 1287 * MAC_TX_SRS_BLOCK 1288 * 1289 * Always called from mac_tx_srs_drain() function. SRS_TX_BLOCKED 1290 * will be set only if srs_tx_woken_up is FALSE. If 1291 * srs_tx_woken_up is TRUE, it indicates that the wakeup arrived 1292 * before we grabbed srs_lock to set SRS_TX_BLOCKED. We need to 1293 * attempt to transmit again and not setting SRS_TX_BLOCKED does 1294 * that. 1295 */ 1296 #define MAC_TX_SRS_BLOCK(srs, mp) { \ 1297 ASSERT(MUTEX_HELD(&(srs)->srs_lock)); \ 1298 if ((srs)->srs_tx.st_woken_up) { \ 1299 (srs)->srs_tx.st_woken_up = B_FALSE; \ 1300 } else { \ 1301 ASSERT(!((srs)->srs_state & SRS_TX_BLOCKED)); \ 1302 (srs)->srs_state |= SRS_TX_BLOCKED; \ 1303 (srs)->srs_tx.st_stat.mts_blockcnt++; \ 1304 } \ 1305 } 1306 1307 /* 1308 * MAC_TX_SRS_TEST_HIWAT 1309 * 1310 * Called before queueing a packet onto Tx SRS to test and set 1311 * SRS_TX_HIWAT if srs_count exceeds srs_tx_hiwat. 1312 */ 1313 #define MAC_TX_SRS_TEST_HIWAT(srs, mp, tail, cnt, sz, cookie) { \ 1314 boolean_t enqueue = 1; \ 1315 \ 1316 if ((srs)->srs_count > (srs)->srs_tx.st_hiwat) { \ 1317 /* \ 1318 * flow-controlled. Store srs in cookie so that it \ 1319 * can be returned as mac_tx_cookie_t to client \ 1320 */ \ 1321 (srs)->srs_state |= SRS_TX_HIWAT; \ 1322 cookie = (mac_tx_cookie_t)srs; \ 1323 (srs)->srs_tx.st_hiwat_cnt++; \ 1324 if ((srs)->srs_count > (srs)->srs_tx.st_max_q_cnt) { \ 1325 /* increment freed stats */ \ 1326 (srs)->srs_tx.st_stat.mts_sdrops += cnt; \ 1327 /* \ 1328 * b_prev may be set to the fanout hint \ 1329 * hence can't use freemsg directly \ 1330 */ \ 1331 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE); \ 1332 DTRACE_PROBE1(tx_queued_hiwat, \ 1333 mac_soft_ring_set_t *, srs); \ 1334 enqueue = 0; \ 1335 } \ 1336 } \ 1337 if (enqueue) \ 1338 MAC_TX_SRS_ENQUEUE_CHAIN(srs, mp, tail, cnt, sz); \ 1339 } 1340 1341 /* Some utility macros */ 1342 #define MAC_SRS_BW_LOCK(srs) \ 1343 if (!(srs->srs_type & SRST_TX)) \ 1344 mutex_enter(&srs->srs_bw->mac_bw_lock); 1345 1346 #define MAC_SRS_BW_UNLOCK(srs) \ 1347 if (!(srs->srs_type & SRST_TX)) \ 1348 mutex_exit(&srs->srs_bw->mac_bw_lock); 1349 1350 #define MAC_TX_SRS_DROP_MESSAGE(srs, mp, cookie) { \ 1351 mac_pkt_drop(NULL, NULL, mp, B_FALSE); \ 1352 /* increment freed stats */ \ 1353 mac_srs->srs_tx.st_stat.mts_sdrops++; \ 1354 cookie = (mac_tx_cookie_t)srs; \ 1355 } 1356 1357 #define MAC_TX_SET_NO_ENQUEUE(srs, mp_chain, ret_mp, cookie) { \ 1358 mac_srs->srs_state |= SRS_TX_WAKEUP_CLIENT; \ 1359 cookie = (mac_tx_cookie_t)srs; \ 1360 *ret_mp = mp_chain; \ 1361 } 1362 1363 /* 1364 * MAC_RX_SRS_TOODEEP 1365 * 1366 * Macro called as part of receive-side processing to determine if handling 1367 * can occur in situ (in the interrupt thread) or if it should be left to a 1368 * worker thread. Note that the constant used to make this determination is 1369 * not entirely made-up, and is a result of some emprical validation. That 1370 * said, the constant is left as a static variable to allow it to be 1371 * dynamically tuned in the field if and as needed. 1372 */ 1373 static uintptr_t mac_rx_srs_stack_needed = 10240; 1374 static uint_t mac_rx_srs_stack_toodeep; 1375 1376 #ifndef STACK_GROWTH_DOWN 1377 #error Downward stack growth assumed. 1378 #endif 1379 1380 #define MAC_RX_SRS_TOODEEP() (STACK_BIAS + (uintptr_t)getfp() - \ 1381 (uintptr_t)curthread->t_stkbase < mac_rx_srs_stack_needed && \ 1382 ++mac_rx_srs_stack_toodeep) 1383 1384 1385 /* 1386 * Drop the rx packet and advance to the next one in the chain. 1387 */ 1388 static void 1389 mac_rx_drop_pkt(mac_soft_ring_set_t *srs, mblk_t *mp) 1390 { 1391 mac_srs_rx_t *srs_rx = &srs->srs_rx; 1392 1393 ASSERT(mp->b_next == NULL); 1394 mutex_enter(&srs->srs_lock); 1395 MAC_UPDATE_SRS_COUNT_LOCKED(srs, 1); 1396 MAC_UPDATE_SRS_SIZE_LOCKED(srs, msgdsize(mp)); 1397 mutex_exit(&srs->srs_lock); 1398 1399 srs_rx->sr_stat.mrs_sdrops++; 1400 freemsg(mp); 1401 } 1402 1403 /* DATAPATH RUNTIME ROUTINES */ 1404 1405 /* 1406 * mac_srs_fire 1407 * 1408 * Timer callback routine for waking up the SRS worker thread. 1409 */ 1410 static void 1411 mac_srs_fire(void *arg) 1412 { 1413 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)arg; 1414 1415 mutex_enter(&mac_srs->srs_lock); 1416 if (mac_srs->srs_tid == NULL) { 1417 mutex_exit(&mac_srs->srs_lock); 1418 return; 1419 } 1420 1421 mac_srs->srs_tid = NULL; 1422 if (!(mac_srs->srs_state & SRS_PROC)) 1423 cv_signal(&mac_srs->srs_async); 1424 1425 mutex_exit(&mac_srs->srs_lock); 1426 } 1427 1428 /* 1429 * 'hint' is fanout_hint (type of uint64_t) which is given by the TCP/IP stack, 1430 * and it is used on the TX path. 1431 */ 1432 #define HASH_HINT(hint) \ 1433 ((hint) ^ ((hint) >> 24) ^ ((hint) >> 16) ^ ((hint) >> 8)) 1434 1435 1436 /* 1437 * hash based on the src address, dst address and the port information. 1438 */ 1439 #define HASH_ADDR(src, dst, ports) \ 1440 (ntohl((src) + (dst)) ^ ((ports) >> 24) ^ ((ports) >> 16) ^ \ 1441 ((ports) >> 8) ^ (ports)) 1442 1443 #define COMPUTE_INDEX(key, sz) (key % sz) 1444 1445 #define FANOUT_ENQUEUE_MP(head, tail, cnt, bw_ctl, sz, sz0, mp) { \ 1446 if ((tail) != NULL) { \ 1447 ASSERT((tail)->b_next == NULL); \ 1448 (tail)->b_next = (mp); \ 1449 } else { \ 1450 ASSERT((head) == NULL); \ 1451 (head) = (mp); \ 1452 } \ 1453 (tail) = (mp); \ 1454 (cnt)++; \ 1455 if ((bw_ctl)) \ 1456 (sz) += (sz0); \ 1457 } 1458 1459 #define MAC_FANOUT_DEFAULT 0 1460 #define MAC_FANOUT_RND_ROBIN 1 1461 int mac_fanout_type = MAC_FANOUT_DEFAULT; 1462 1463 #define MAX_SR_TYPES 3 1464 /* fanout types for port based hashing */ 1465 enum pkt_type { 1466 V4_TCP = 0, 1467 V4_UDP, 1468 OTH, 1469 UNDEF 1470 }; 1471 1472 /* 1473 * Pair of local and remote ports in the transport header 1474 */ 1475 #define PORTS_SIZE 4 1476 1477 /* 1478 * mac_rx_srs_proto_fanout 1479 * 1480 * This routine delivers packets destined to an SRS into one of the 1481 * protocol soft rings. 1482 * 1483 * Given a chain of packets we need to split it up into multiple sub chains 1484 * destined into TCP, UDP or OTH soft ring. Instead of entering 1485 * the soft ring one packet at a time, we want to enter it in the form of a 1486 * chain otherwise we get this start/stop behaviour where the worker thread 1487 * goes to sleep and then next packets comes in forcing it to wake up etc. 1488 */ 1489 static void 1490 mac_rx_srs_proto_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head) 1491 { 1492 struct ether_header *ehp; 1493 struct ether_vlan_header *evhp; 1494 uint32_t sap; 1495 ipha_t *ipha; 1496 uint8_t *dstaddr; 1497 size_t hdrsize; 1498 mblk_t *mp; 1499 mblk_t *headmp[MAX_SR_TYPES]; 1500 mblk_t *tailmp[MAX_SR_TYPES]; 1501 int cnt[MAX_SR_TYPES]; 1502 size_t sz[MAX_SR_TYPES]; 1503 size_t sz1; 1504 boolean_t bw_ctl; 1505 boolean_t hw_classified; 1506 boolean_t dls_bypass; 1507 boolean_t is_ether; 1508 boolean_t is_unicast; 1509 enum pkt_type type; 1510 mac_client_impl_t *mcip = mac_srs->srs_mcip; 1511 1512 is_ether = (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER); 1513 bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0); 1514 1515 /* 1516 * If we don't have a Rx ring, S/W classification would have done 1517 * its job and its a packet meant for us. If we were polling on 1518 * the default ring (i.e. there was a ring assigned to this SRS), 1519 * then we need to make sure that the mac address really belongs 1520 * to us. 1521 */ 1522 hw_classified = mac_srs->srs_ring != NULL && 1523 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER; 1524 1525 /* 1526 * Special clients (eg. VLAN, non ether, etc) need DLS 1527 * processing in the Rx path. SRST_DLS_BYPASS will be clear for 1528 * such SRSs. Another way of disabling bypass is to set the 1529 * MCIS_RX_BYPASS_DISABLE flag. 1530 */ 1531 dls_bypass = ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) && 1532 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0); 1533 1534 bzero(headmp, MAX_SR_TYPES * sizeof (mblk_t *)); 1535 bzero(tailmp, MAX_SR_TYPES * sizeof (mblk_t *)); 1536 bzero(cnt, MAX_SR_TYPES * sizeof (int)); 1537 bzero(sz, MAX_SR_TYPES * sizeof (size_t)); 1538 1539 /* 1540 * We got a chain from SRS that we need to send to the soft rings. 1541 * Since squeues for TCP & IPv4 sap poll their soft rings (for 1542 * performance reasons), we need to separate out v4_tcp, v4_udp 1543 * and the rest goes in other. 1544 */ 1545 while (head != NULL) { 1546 mp = head; 1547 head = head->b_next; 1548 mp->b_next = NULL; 1549 1550 type = OTH; 1551 sz1 = (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp); 1552 1553 if (is_ether) { 1554 /* 1555 * At this point we can be sure the packet at least 1556 * has an ether header. 1557 */ 1558 if (sz1 < sizeof (struct ether_header)) { 1559 mac_rx_drop_pkt(mac_srs, mp); 1560 continue; 1561 } 1562 ehp = (struct ether_header *)mp->b_rptr; 1563 1564 /* 1565 * Determine if this is a VLAN or non-VLAN packet. 1566 */ 1567 if ((sap = ntohs(ehp->ether_type)) == VLAN_TPID) { 1568 evhp = (struct ether_vlan_header *)mp->b_rptr; 1569 sap = ntohs(evhp->ether_type); 1570 hdrsize = sizeof (struct ether_vlan_header); 1571 /* 1572 * Check if the VID of the packet, if any, 1573 * belongs to this client. 1574 */ 1575 if (!mac_client_check_flow_vid(mcip, 1576 VLAN_ID(ntohs(evhp->ether_tci)))) { 1577 mac_rx_drop_pkt(mac_srs, mp); 1578 continue; 1579 } 1580 } else { 1581 hdrsize = sizeof (struct ether_header); 1582 } 1583 is_unicast = 1584 ((((uint8_t *)&ehp->ether_dhost)[0] & 0x01) == 0); 1585 dstaddr = (uint8_t *)&ehp->ether_dhost; 1586 } else { 1587 mac_header_info_t mhi; 1588 1589 if (mac_header_info((mac_handle_t)mcip->mci_mip, 1590 mp, &mhi) != 0) { 1591 mac_rx_drop_pkt(mac_srs, mp); 1592 continue; 1593 } 1594 hdrsize = mhi.mhi_hdrsize; 1595 sap = mhi.mhi_bindsap; 1596 is_unicast = (mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST); 1597 dstaddr = (uint8_t *)mhi.mhi_daddr; 1598 } 1599 1600 if (!dls_bypass) { 1601 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], 1602 cnt[type], bw_ctl, sz[type], sz1, mp); 1603 continue; 1604 } 1605 1606 if (sap == ETHERTYPE_IP) { 1607 /* 1608 * If we are H/W classified, but we have promisc 1609 * on, then we need to check for the unicast address. 1610 */ 1611 if (hw_classified && mcip->mci_promisc_list != NULL) { 1612 mac_address_t *map; 1613 1614 rw_enter(&mcip->mci_rw_lock, RW_READER); 1615 map = mcip->mci_unicast; 1616 if (bcmp(dstaddr, map->ma_addr, 1617 map->ma_len) == 0) 1618 type = UNDEF; 1619 rw_exit(&mcip->mci_rw_lock); 1620 } else if (is_unicast) { 1621 type = UNDEF; 1622 } 1623 } 1624 1625 /* 1626 * This needs to become a contract with the driver for 1627 * the fast path. 1628 * 1629 * In the normal case the packet will have at least the L2 1630 * header and the IP + Transport header in the same mblk. 1631 * This is usually the case when the NIC driver sends up 1632 * the packet. This is also true when the stack generates 1633 * a packet that is looped back and when the stack uses the 1634 * fastpath mechanism. The normal case is optimized for 1635 * performance and may bypass DLS. All other cases go through 1636 * the 'OTH' type path without DLS bypass. 1637 */ 1638 1639 ipha = (ipha_t *)(mp->b_rptr + hdrsize); 1640 if ((type != OTH) && MBLK_RX_FANOUT_SLOWPATH(mp, ipha)) 1641 type = OTH; 1642 1643 if (type == OTH) { 1644 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], 1645 cnt[type], bw_ctl, sz[type], sz1, mp); 1646 continue; 1647 } 1648 1649 ASSERT(type == UNDEF); 1650 /* 1651 * We look for at least 4 bytes past the IP header to get 1652 * the port information. If we get an IP fragment, we don't 1653 * have the port information, and we use just the protocol 1654 * information. 1655 */ 1656 switch (ipha->ipha_protocol) { 1657 case IPPROTO_TCP: 1658 type = V4_TCP; 1659 mp->b_rptr += hdrsize; 1660 break; 1661 case IPPROTO_UDP: 1662 type = V4_UDP; 1663 mp->b_rptr += hdrsize; 1664 break; 1665 default: 1666 type = OTH; 1667 break; 1668 } 1669 1670 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], cnt[type], 1671 bw_ctl, sz[type], sz1, mp); 1672 } 1673 1674 for (type = V4_TCP; type < UNDEF; type++) { 1675 if (headmp[type] != NULL) { 1676 mac_soft_ring_t *softring; 1677 1678 ASSERT(tailmp[type]->b_next == NULL); 1679 switch (type) { 1680 case V4_TCP: 1681 softring = mac_srs->srs_tcp_soft_rings[0]; 1682 break; 1683 case V4_UDP: 1684 softring = mac_srs->srs_udp_soft_rings[0]; 1685 break; 1686 case OTH: 1687 softring = mac_srs->srs_oth_soft_rings[0]; 1688 } 1689 mac_rx_soft_ring_process(mcip, softring, 1690 headmp[type], tailmp[type], cnt[type], sz[type]); 1691 } 1692 } 1693 } 1694 1695 int fanout_unaligned = 0; 1696 1697 /* 1698 * mac_rx_srs_long_fanout 1699 * 1700 * The fanout routine for VLANs, and for anything else that isn't performing 1701 * explicit dls bypass. Returns -1 on an error (drop the packet due to a 1702 * malformed packet), 0 on success, with values written in *indx and *type. 1703 */ 1704 static int 1705 mac_rx_srs_long_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *mp, 1706 uint32_t sap, size_t hdrsize, enum pkt_type *type, uint_t *indx) 1707 { 1708 ip6_t *ip6h; 1709 ipha_t *ipha; 1710 uint8_t *whereptr; 1711 uint_t hash; 1712 uint16_t remlen; 1713 uint8_t nexthdr; 1714 uint16_t hdr_len; 1715 uint32_t src_val, dst_val; 1716 boolean_t modifiable = B_TRUE; 1717 boolean_t v6; 1718 1719 ASSERT(MBLKL(mp) >= hdrsize); 1720 1721 if (sap == ETHERTYPE_IPV6) { 1722 v6 = B_TRUE; 1723 hdr_len = IPV6_HDR_LEN; 1724 } else if (sap == ETHERTYPE_IP) { 1725 v6 = B_FALSE; 1726 hdr_len = IP_SIMPLE_HDR_LENGTH; 1727 } else { 1728 *indx = 0; 1729 *type = OTH; 1730 return (0); 1731 } 1732 1733 ip6h = (ip6_t *)(mp->b_rptr + hdrsize); 1734 ipha = (ipha_t *)ip6h; 1735 1736 if ((uint8_t *)ip6h == mp->b_wptr) { 1737 /* 1738 * The first mblk_t only includes the mac header. 1739 * Note that it is safe to change the mp pointer here, 1740 * as the subsequent operation does not assume mp 1741 * points to the start of the mac header. 1742 */ 1743 mp = mp->b_cont; 1744 1745 /* 1746 * Make sure the IP header points to an entire one. 1747 */ 1748 if (mp == NULL) 1749 return (-1); 1750 1751 if (MBLKL(mp) < hdr_len) { 1752 modifiable = (DB_REF(mp) == 1); 1753 1754 if (modifiable && !pullupmsg(mp, hdr_len)) 1755 return (-1); 1756 } 1757 1758 ip6h = (ip6_t *)mp->b_rptr; 1759 ipha = (ipha_t *)ip6h; 1760 } 1761 1762 if (!modifiable || !(OK_32PTR((char *)ip6h)) || 1763 ((uint8_t *)ip6h + hdr_len > mp->b_wptr)) { 1764 /* 1765 * If either the IP header is not aligned, or it does not hold 1766 * the complete simple structure (a pullupmsg() is not an 1767 * option since it would result in an unaligned IP header), 1768 * fanout to the default ring. 1769 * 1770 * Note that this may cause packet reordering. 1771 */ 1772 *indx = 0; 1773 *type = OTH; 1774 fanout_unaligned++; 1775 return (0); 1776 } 1777 1778 /* 1779 * Extract next-header, full header length, and source-hash value 1780 * using v4/v6 specific fields. 1781 */ 1782 if (v6) { 1783 remlen = ntohs(ip6h->ip6_plen); 1784 nexthdr = ip6h->ip6_nxt; 1785 src_val = V4_PART_OF_V6(ip6h->ip6_src); 1786 dst_val = V4_PART_OF_V6(ip6h->ip6_dst); 1787 /* 1788 * Do src based fanout if below tunable is set to B_TRUE or 1789 * when mac_ip_hdr_length_v6() fails because of malformed 1790 * packets or because mblks need to be concatenated using 1791 * pullupmsg(). 1792 * 1793 * Perform a version check to prevent parsing weirdness... 1794 */ 1795 if (IPH_HDR_VERSION(ip6h) != IPV6_VERSION || 1796 !mac_ip_hdr_length_v6(ip6h, mp->b_wptr, &hdr_len, &nexthdr, 1797 NULL)) { 1798 goto src_dst_based_fanout; 1799 } 1800 } else { 1801 hdr_len = IPH_HDR_LENGTH(ipha); 1802 remlen = ntohs(ipha->ipha_length) - hdr_len; 1803 nexthdr = ipha->ipha_protocol; 1804 src_val = (uint32_t)ipha->ipha_src; 1805 dst_val = (uint32_t)ipha->ipha_dst; 1806 /* 1807 * Catch IPv4 fragment case here. IPv6 has nexthdr == FRAG 1808 * for its equivalent case. 1809 */ 1810 if ((ntohs(ipha->ipha_fragment_offset_and_flags) & 1811 (IPH_MF | IPH_OFFSET)) != 0) { 1812 goto src_dst_based_fanout; 1813 } 1814 } 1815 if (remlen < MIN_EHDR_LEN) 1816 return (-1); 1817 whereptr = (uint8_t *)ip6h + hdr_len; 1818 1819 /* If the transport is one of below, we do port/SPI based fanout */ 1820 switch (nexthdr) { 1821 case IPPROTO_TCP: 1822 case IPPROTO_UDP: 1823 case IPPROTO_SCTP: 1824 case IPPROTO_ESP: 1825 /* 1826 * If the ports or SPI in the transport header is not part of 1827 * the mblk, do src_based_fanout, instead of calling 1828 * pullupmsg(). 1829 */ 1830 if (mp->b_cont == NULL || whereptr + PORTS_SIZE <= mp->b_wptr) 1831 break; /* out of switch... */ 1832 /* FALLTHRU */ 1833 default: 1834 goto src_dst_based_fanout; 1835 } 1836 1837 switch (nexthdr) { 1838 case IPPROTO_TCP: 1839 hash = HASH_ADDR(src_val, dst_val, *(uint32_t *)whereptr); 1840 *indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count); 1841 *type = OTH; 1842 break; 1843 case IPPROTO_UDP: 1844 case IPPROTO_SCTP: 1845 case IPPROTO_ESP: 1846 if (mac_fanout_type == MAC_FANOUT_DEFAULT) { 1847 hash = HASH_ADDR(src_val, dst_val, 1848 *(uint32_t *)whereptr); 1849 *indx = COMPUTE_INDEX(hash, 1850 mac_srs->srs_udp_ring_count); 1851 } else { 1852 *indx = mac_srs->srs_ind % mac_srs->srs_udp_ring_count; 1853 mac_srs->srs_ind++; 1854 } 1855 *type = OTH; 1856 break; 1857 } 1858 return (0); 1859 1860 src_dst_based_fanout: 1861 hash = HASH_ADDR(src_val, dst_val, (uint32_t)0); 1862 *indx = COMPUTE_INDEX(hash, mac_srs->srs_oth_ring_count); 1863 *type = OTH; 1864 return (0); 1865 } 1866 1867 /* 1868 * mac_rx_srs_fanout 1869 * 1870 * This routine delivers packets destined to an SRS into a soft ring member 1871 * of the set. 1872 * 1873 * Given a chain of packets we need to split it up into multiple sub chains 1874 * destined for one of the TCP, UDP or OTH soft rings. Instead of entering 1875 * the soft ring one packet at a time, we want to enter it in the form of a 1876 * chain otherwise we get this start/stop behaviour where the worker thread 1877 * goes to sleep and then next packets comes in forcing it to wake up etc. 1878 * 1879 * Note: 1880 * Since we know what is the maximum fanout possible, we create a 2D array 1881 * of 'softring types * MAX_SR_FANOUT' for the head, tail, cnt and sz 1882 * variables so that we can enter the softrings with chain. We need the 1883 * MAX_SR_FANOUT so we can allocate the arrays on the stack (a kmem_alloc 1884 * for each packet would be expensive). If we ever want to have the 1885 * ability to have unlimited fanout, we should probably declare a head, 1886 * tail, cnt, sz with each soft ring (a data struct which contains a softring 1887 * along with these members) and create an array of this uber struct so we 1888 * don't have to do kmem_alloc. 1889 */ 1890 int fanout_oth1 = 0; 1891 int fanout_oth2 = 0; 1892 int fanout_oth3 = 0; 1893 int fanout_oth4 = 0; 1894 int fanout_oth5 = 0; 1895 1896 static void 1897 mac_rx_srs_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head) 1898 { 1899 struct ether_header *ehp; 1900 struct ether_vlan_header *evhp; 1901 uint32_t sap; 1902 ipha_t *ipha; 1903 uint8_t *dstaddr; 1904 uint_t indx; 1905 size_t ports_offset; 1906 size_t ipha_len; 1907 size_t hdrsize; 1908 uint_t hash; 1909 mblk_t *mp; 1910 mblk_t *headmp[MAX_SR_TYPES][MAX_SR_FANOUT]; 1911 mblk_t *tailmp[MAX_SR_TYPES][MAX_SR_FANOUT]; 1912 int cnt[MAX_SR_TYPES][MAX_SR_FANOUT]; 1913 size_t sz[MAX_SR_TYPES][MAX_SR_FANOUT]; 1914 size_t sz1; 1915 boolean_t bw_ctl; 1916 boolean_t hw_classified; 1917 boolean_t dls_bypass; 1918 boolean_t is_ether; 1919 boolean_t is_unicast; 1920 int fanout_cnt; 1921 enum pkt_type type; 1922 mac_client_impl_t *mcip = mac_srs->srs_mcip; 1923 1924 is_ether = (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER); 1925 bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0); 1926 1927 /* 1928 * If we don't have a Rx ring, S/W classification would have done 1929 * its job and its a packet meant for us. If we were polling on 1930 * the default ring (i.e. there was a ring assigned to this SRS), 1931 * then we need to make sure that the mac address really belongs 1932 * to us. 1933 */ 1934 hw_classified = mac_srs->srs_ring != NULL && 1935 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER; 1936 1937 /* 1938 * Special clients (eg. VLAN, non ether, etc) need DLS 1939 * processing in the Rx path. SRST_DLS_BYPASS will be clear for 1940 * such SRSs. Another way of disabling bypass is to set the 1941 * MCIS_RX_BYPASS_DISABLE flag. 1942 */ 1943 dls_bypass = ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) && 1944 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0); 1945 1946 /* 1947 * Since the softrings are never destroyed and we always 1948 * create equal number of softrings for TCP, UDP and rest, 1949 * its OK to check one of them for count and use it without 1950 * any lock. In future, if soft rings get destroyed because 1951 * of reduction in fanout, we will need to ensure that happens 1952 * behind the SRS_PROC. 1953 */ 1954 fanout_cnt = mac_srs->srs_tcp_ring_count; 1955 1956 bzero(headmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *)); 1957 bzero(tailmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *)); 1958 bzero(cnt, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (int)); 1959 bzero(sz, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (size_t)); 1960 1961 /* 1962 * We got a chain from SRS that we need to send to the soft rings. 1963 * Since squeues for TCP & IPv4 sap poll their soft rings (for 1964 * performance reasons), we need to separate out v4_tcp, v4_udp 1965 * and the rest goes in other. 1966 */ 1967 while (head != NULL) { 1968 mp = head; 1969 head = head->b_next; 1970 mp->b_next = NULL; 1971 1972 type = OTH; 1973 sz1 = (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp); 1974 1975 if (is_ether) { 1976 /* 1977 * At this point we can be sure the packet at least 1978 * has an ether header. 1979 */ 1980 if (sz1 < sizeof (struct ether_header)) { 1981 mac_rx_drop_pkt(mac_srs, mp); 1982 continue; 1983 } 1984 ehp = (struct ether_header *)mp->b_rptr; 1985 1986 /* 1987 * Determine if this is a VLAN or non-VLAN packet. 1988 */ 1989 if ((sap = ntohs(ehp->ether_type)) == VLAN_TPID) { 1990 evhp = (struct ether_vlan_header *)mp->b_rptr; 1991 sap = ntohs(evhp->ether_type); 1992 hdrsize = sizeof (struct ether_vlan_header); 1993 /* 1994 * Check if the VID of the packet, if any, 1995 * belongs to this client. 1996 */ 1997 if (!mac_client_check_flow_vid(mcip, 1998 VLAN_ID(ntohs(evhp->ether_tci)))) { 1999 mac_rx_drop_pkt(mac_srs, mp); 2000 continue; 2001 } 2002 } else { 2003 hdrsize = sizeof (struct ether_header); 2004 } 2005 is_unicast = 2006 ((((uint8_t *)&ehp->ether_dhost)[0] & 0x01) == 0); 2007 dstaddr = (uint8_t *)&ehp->ether_dhost; 2008 } else { 2009 mac_header_info_t mhi; 2010 2011 if (mac_header_info((mac_handle_t)mcip->mci_mip, 2012 mp, &mhi) != 0) { 2013 mac_rx_drop_pkt(mac_srs, mp); 2014 continue; 2015 } 2016 hdrsize = mhi.mhi_hdrsize; 2017 sap = mhi.mhi_bindsap; 2018 is_unicast = (mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST); 2019 dstaddr = (uint8_t *)mhi.mhi_daddr; 2020 } 2021 2022 if (!dls_bypass) { 2023 if (mac_rx_srs_long_fanout(mac_srs, mp, sap, 2024 hdrsize, &type, &indx) == -1) { 2025 mac_rx_drop_pkt(mac_srs, mp); 2026 continue; 2027 } 2028 2029 FANOUT_ENQUEUE_MP(headmp[type][indx], 2030 tailmp[type][indx], cnt[type][indx], bw_ctl, 2031 sz[type][indx], sz1, mp); 2032 continue; 2033 } 2034 2035 2036 /* 2037 * If we are using the default Rx ring where H/W or S/W 2038 * classification has not happened, we need to verify if 2039 * this unicast packet really belongs to us. 2040 */ 2041 if (sap == ETHERTYPE_IP) { 2042 /* 2043 * If we are H/W classified, but we have promisc 2044 * on, then we need to check for the unicast address. 2045 */ 2046 if (hw_classified && mcip->mci_promisc_list != NULL) { 2047 mac_address_t *map; 2048 2049 rw_enter(&mcip->mci_rw_lock, RW_READER); 2050 map = mcip->mci_unicast; 2051 if (bcmp(dstaddr, map->ma_addr, 2052 map->ma_len) == 0) 2053 type = UNDEF; 2054 rw_exit(&mcip->mci_rw_lock); 2055 } else if (is_unicast) { 2056 type = UNDEF; 2057 } 2058 } 2059 2060 /* 2061 * This needs to become a contract with the driver for 2062 * the fast path. 2063 */ 2064 2065 ipha = (ipha_t *)(mp->b_rptr + hdrsize); 2066 if ((type != OTH) && MBLK_RX_FANOUT_SLOWPATH(mp, ipha)) { 2067 type = OTH; 2068 fanout_oth1++; 2069 } 2070 2071 if (type != OTH) { 2072 uint16_t frag_offset_flags; 2073 2074 switch (ipha->ipha_protocol) { 2075 case IPPROTO_TCP: 2076 case IPPROTO_UDP: 2077 case IPPROTO_SCTP: 2078 case IPPROTO_ESP: 2079 ipha_len = IPH_HDR_LENGTH(ipha); 2080 if ((uchar_t *)ipha + ipha_len + PORTS_SIZE > 2081 mp->b_wptr) { 2082 type = OTH; 2083 break; 2084 } 2085 frag_offset_flags = 2086 ntohs(ipha->ipha_fragment_offset_and_flags); 2087 if ((frag_offset_flags & 2088 (IPH_MF | IPH_OFFSET)) != 0) { 2089 type = OTH; 2090 fanout_oth3++; 2091 break; 2092 } 2093 ports_offset = hdrsize + ipha_len; 2094 break; 2095 default: 2096 type = OTH; 2097 fanout_oth4++; 2098 break; 2099 } 2100 } 2101 2102 if (type == OTH) { 2103 if (mac_rx_srs_long_fanout(mac_srs, mp, sap, 2104 hdrsize, &type, &indx) == -1) { 2105 mac_rx_drop_pkt(mac_srs, mp); 2106 continue; 2107 } 2108 2109 FANOUT_ENQUEUE_MP(headmp[type][indx], 2110 tailmp[type][indx], cnt[type][indx], bw_ctl, 2111 sz[type][indx], sz1, mp); 2112 continue; 2113 } 2114 2115 ASSERT(type == UNDEF); 2116 2117 /* 2118 * XXX-Sunay: We should hold srs_lock since ring_count 2119 * below can change. But if we are always called from 2120 * mac_rx_srs_drain and SRS_PROC is set, then we can 2121 * enforce that ring_count can't be changed i.e. 2122 * to change fanout type or ring count, the calling 2123 * thread needs to be behind SRS_PROC. 2124 */ 2125 switch (ipha->ipha_protocol) { 2126 case IPPROTO_TCP: 2127 /* 2128 * Note that for ESP, we fanout on SPI and it is at the 2129 * same offset as the 2x16-bit ports. So it is clumped 2130 * along with TCP, UDP and SCTP. 2131 */ 2132 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst, 2133 *(uint32_t *)(mp->b_rptr + ports_offset)); 2134 indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count); 2135 type = V4_TCP; 2136 mp->b_rptr += hdrsize; 2137 break; 2138 case IPPROTO_UDP: 2139 case IPPROTO_SCTP: 2140 case IPPROTO_ESP: 2141 if (mac_fanout_type == MAC_FANOUT_DEFAULT) { 2142 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst, 2143 *(uint32_t *)(mp->b_rptr + ports_offset)); 2144 indx = COMPUTE_INDEX(hash, 2145 mac_srs->srs_udp_ring_count); 2146 } else { 2147 indx = mac_srs->srs_ind % 2148 mac_srs->srs_udp_ring_count; 2149 mac_srs->srs_ind++; 2150 } 2151 type = V4_UDP; 2152 mp->b_rptr += hdrsize; 2153 break; 2154 default: 2155 indx = 0; 2156 type = OTH; 2157 } 2158 2159 FANOUT_ENQUEUE_MP(headmp[type][indx], tailmp[type][indx], 2160 cnt[type][indx], bw_ctl, sz[type][indx], sz1, mp); 2161 } 2162 2163 for (type = V4_TCP; type < UNDEF; type++) { 2164 int i; 2165 2166 for (i = 0; i < fanout_cnt; i++) { 2167 if (headmp[type][i] != NULL) { 2168 mac_soft_ring_t *softring; 2169 2170 ASSERT(tailmp[type][i]->b_next == NULL); 2171 switch (type) { 2172 case V4_TCP: 2173 softring = 2174 mac_srs->srs_tcp_soft_rings[i]; 2175 break; 2176 case V4_UDP: 2177 softring = 2178 mac_srs->srs_udp_soft_rings[i]; 2179 break; 2180 case OTH: 2181 softring = 2182 mac_srs->srs_oth_soft_rings[i]; 2183 break; 2184 } 2185 mac_rx_soft_ring_process(mcip, 2186 softring, headmp[type][i], tailmp[type][i], 2187 cnt[type][i], sz[type][i]); 2188 } 2189 } 2190 } 2191 } 2192 2193 #define SRS_BYTES_TO_PICKUP 150000 2194 ssize_t max_bytes_to_pickup = SRS_BYTES_TO_PICKUP; 2195 2196 /* 2197 * mac_rx_srs_poll_ring 2198 * 2199 * This SRS Poll thread uses this routine to poll the underlying hardware 2200 * Rx ring to get a chain of packets. It can inline process that chain 2201 * if mac_latency_optimize is set (default) or signal the SRS worker thread 2202 * to do the remaining processing. 2203 * 2204 * Since packets come in the system via interrupt or poll path, we also 2205 * update the stats and deal with promiscous clients here. 2206 */ 2207 void 2208 mac_rx_srs_poll_ring(mac_soft_ring_set_t *mac_srs) 2209 { 2210 kmutex_t *lock = &mac_srs->srs_lock; 2211 kcondvar_t *async = &mac_srs->srs_cv; 2212 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2213 mblk_t *head, *tail, *mp; 2214 callb_cpr_t cprinfo; 2215 ssize_t bytes_to_pickup; 2216 size_t sz; 2217 int count; 2218 mac_client_impl_t *smcip; 2219 2220 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "mac_srs_poll"); 2221 mutex_enter(lock); 2222 2223 start: 2224 for (;;) { 2225 if (mac_srs->srs_state & SRS_PAUSE) 2226 goto done; 2227 2228 CALLB_CPR_SAFE_BEGIN(&cprinfo); 2229 cv_wait(async, lock); 2230 CALLB_CPR_SAFE_END(&cprinfo, lock); 2231 2232 if (mac_srs->srs_state & SRS_PAUSE) 2233 goto done; 2234 2235 check_again: 2236 if (mac_srs->srs_type & SRST_BW_CONTROL) { 2237 /* 2238 * We pick as many bytes as we are allowed to queue. 2239 * Its possible that we will exceed the total 2240 * packets queued in case this SRS is part of the 2241 * Rx ring group since > 1 poll thread can be pulling 2242 * upto the max allowed packets at the same time 2243 * but that should be OK. 2244 */ 2245 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2246 bytes_to_pickup = 2247 mac_srs->srs_bw->mac_bw_drop_threshold - 2248 mac_srs->srs_bw->mac_bw_sz; 2249 /* 2250 * We shouldn't have been signalled if we 2251 * have 0 or less bytes to pick but since 2252 * some of the bytes accounting is driver 2253 * dependant, we do the safety check. 2254 */ 2255 if (bytes_to_pickup < 0) 2256 bytes_to_pickup = 0; 2257 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2258 } else { 2259 /* 2260 * ToDO: Need to change the polling API 2261 * to add a packet count and a flag which 2262 * tells the driver whether we want packets 2263 * based on a count, or bytes, or all the 2264 * packets queued in the driver/HW. This 2265 * way, we never have to check the limits 2266 * on poll path. We truly let only as many 2267 * packets enter the system as we are willing 2268 * to process or queue. 2269 * 2270 * Something along the lines of 2271 * pkts_to_pickup = mac_soft_ring_max_q_cnt - 2272 * mac_srs->srs_poll_pkt_cnt 2273 */ 2274 2275 /* 2276 * Since we are not doing B/W control, pick 2277 * as many packets as allowed. 2278 */ 2279 bytes_to_pickup = max_bytes_to_pickup; 2280 } 2281 2282 /* Poll the underlying Hardware */ 2283 mutex_exit(lock); 2284 head = MAC_HWRING_POLL(mac_srs->srs_ring, (int)bytes_to_pickup); 2285 mutex_enter(lock); 2286 2287 ASSERT((mac_srs->srs_state & SRS_POLL_THR_OWNER) == 2288 SRS_POLL_THR_OWNER); 2289 2290 mp = tail = head; 2291 count = 0; 2292 sz = 0; 2293 while (mp != NULL) { 2294 tail = mp; 2295 sz += msgdsize(mp); 2296 mp = mp->b_next; 2297 count++; 2298 } 2299 2300 if (head != NULL) { 2301 tail->b_next = NULL; 2302 smcip = mac_srs->srs_mcip; 2303 2304 SRS_RX_STAT_UPDATE(mac_srs, pollbytes, sz); 2305 SRS_RX_STAT_UPDATE(mac_srs, pollcnt, count); 2306 2307 /* 2308 * If there are any promiscuous mode callbacks 2309 * defined for this MAC client, pass them a copy 2310 * if appropriate and also update the counters. 2311 */ 2312 if (smcip != NULL) { 2313 if (smcip->mci_mip->mi_promisc_list != NULL) { 2314 mutex_exit(lock); 2315 mac_promisc_dispatch(smcip->mci_mip, 2316 head, NULL); 2317 mutex_enter(lock); 2318 } 2319 } 2320 if (mac_srs->srs_type & SRST_BW_CONTROL) { 2321 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2322 mac_srs->srs_bw->mac_bw_polled += sz; 2323 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2324 } 2325 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, 2326 count, sz); 2327 if (count <= 10) 2328 srs_rx->sr_stat.mrs_chaincntundr10++; 2329 else if (count > 10 && count <= 50) 2330 srs_rx->sr_stat.mrs_chaincnt10to50++; 2331 else 2332 srs_rx->sr_stat.mrs_chaincntover50++; 2333 } 2334 2335 /* 2336 * We are guaranteed that SRS_PROC will be set if we 2337 * are here. Also, poll thread gets to run only if 2338 * the drain was being done by a worker thread although 2339 * its possible that worker thread is still running 2340 * and poll thread was sent down to keep the pipeline 2341 * going instead of doing a complete drain and then 2342 * trying to poll the NIC. 2343 * 2344 * So we need to check SRS_WORKER flag to make sure 2345 * that the worker thread is not processing the queue 2346 * in parallel to us. The flags and conditions are 2347 * protected by the srs_lock to prevent any race. We 2348 * ensure that we don't drop the srs_lock from now 2349 * till the end and similarly we don't drop the srs_lock 2350 * in mac_rx_srs_drain() till similar condition check 2351 * are complete. The mac_rx_srs_drain() needs to ensure 2352 * that SRS_WORKER flag remains set as long as its 2353 * processing the queue. 2354 */ 2355 if (!(mac_srs->srs_state & SRS_WORKER) && 2356 (mac_srs->srs_first != NULL)) { 2357 /* 2358 * We have packets to process and worker thread 2359 * is not running. Check to see if poll thread is 2360 * allowed to process. 2361 */ 2362 if (mac_srs->srs_state & SRS_LATENCY_OPT) { 2363 mac_srs->srs_drain_func(mac_srs, SRS_POLL_PROC); 2364 if (!(mac_srs->srs_state & SRS_PAUSE) && 2365 srs_rx->sr_poll_pkt_cnt <= 2366 srs_rx->sr_lowat) { 2367 srs_rx->sr_poll_again++; 2368 goto check_again; 2369 } 2370 /* 2371 * We are already above low water mark 2372 * so stay in the polling mode but no 2373 * need to poll. Once we dip below 2374 * the polling threshold, the processing 2375 * thread (soft ring) will signal us 2376 * to poll again (MAC_UPDATE_SRS_COUNT) 2377 */ 2378 srs_rx->sr_poll_drain_no_poll++; 2379 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS); 2380 /* 2381 * In B/W control case, its possible 2382 * that the backlog built up due to 2383 * B/W limit being reached and packets 2384 * are queued only in SRS. In this case, 2385 * we should schedule worker thread 2386 * since no one else will wake us up. 2387 */ 2388 if ((mac_srs->srs_type & SRST_BW_CONTROL) && 2389 (mac_srs->srs_tid == NULL)) { 2390 mac_srs->srs_tid = 2391 timeout(mac_srs_fire, mac_srs, 1); 2392 srs_rx->sr_poll_worker_wakeup++; 2393 } 2394 } else { 2395 /* 2396 * Wakeup the worker thread for more processing. 2397 * We optimize for throughput in this case. 2398 */ 2399 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS); 2400 MAC_SRS_WORKER_WAKEUP(mac_srs); 2401 srs_rx->sr_poll_sig_worker++; 2402 } 2403 } else if ((mac_srs->srs_first == NULL) && 2404 !(mac_srs->srs_state & SRS_WORKER)) { 2405 /* 2406 * There is nothing queued in SRS and 2407 * no worker thread running. Plus we 2408 * didn't get anything from the H/W 2409 * as well (head == NULL); 2410 */ 2411 ASSERT(head == NULL); 2412 mac_srs->srs_state &= 2413 ~(SRS_PROC|SRS_GET_PKTS); 2414 2415 /* 2416 * If we have a packets in soft ring, don't allow 2417 * more packets to come into this SRS by keeping the 2418 * interrupts off but not polling the H/W. The 2419 * poll thread will get signaled as soon as 2420 * srs_poll_pkt_cnt dips below poll threshold. 2421 */ 2422 if (srs_rx->sr_poll_pkt_cnt == 0) { 2423 srs_rx->sr_poll_intr_enable++; 2424 MAC_SRS_POLLING_OFF(mac_srs); 2425 } else { 2426 /* 2427 * We know nothing is queued in SRS 2428 * since we are here after checking 2429 * srs_first is NULL. The backlog 2430 * is entirely due to packets queued 2431 * in Soft ring which will wake us up 2432 * and get the interface out of polling 2433 * mode once the backlog dips below 2434 * sr_poll_thres. 2435 */ 2436 srs_rx->sr_poll_no_poll++; 2437 } 2438 } else { 2439 /* 2440 * Worker thread is already running. 2441 * Nothing much to do. If the polling 2442 * was enabled, worker thread will deal 2443 * with that. 2444 */ 2445 mac_srs->srs_state &= ~SRS_GET_PKTS; 2446 srs_rx->sr_poll_goto_sleep++; 2447 } 2448 } 2449 done: 2450 mac_srs->srs_state |= SRS_POLL_THR_QUIESCED; 2451 cv_signal(&mac_srs->srs_async); 2452 /* 2453 * If this is a temporary quiesce then wait for the restart signal 2454 * from the srs worker. Then clear the flags and signal the srs worker 2455 * to ensure a positive handshake and go back to start. 2456 */ 2457 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_POLL_THR_RESTART))) 2458 cv_wait(async, lock); 2459 if (mac_srs->srs_state & SRS_POLL_THR_RESTART) { 2460 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED)); 2461 mac_srs->srs_state &= 2462 ~(SRS_POLL_THR_QUIESCED | SRS_POLL_THR_RESTART); 2463 cv_signal(&mac_srs->srs_async); 2464 goto start; 2465 } else { 2466 mac_srs->srs_state |= SRS_POLL_THR_EXITED; 2467 cv_signal(&mac_srs->srs_async); 2468 CALLB_CPR_EXIT(&cprinfo); 2469 thread_exit(); 2470 } 2471 } 2472 2473 /* 2474 * mac_srs_pick_chain 2475 * 2476 * In Bandwidth control case, checks how many packets can be processed 2477 * and return them in a sub chain. 2478 */ 2479 static mblk_t * 2480 mac_srs_pick_chain(mac_soft_ring_set_t *mac_srs, mblk_t **chain_tail, 2481 size_t *chain_sz, int *chain_cnt) 2482 { 2483 mblk_t *head = NULL; 2484 mblk_t *tail = NULL; 2485 size_t sz; 2486 size_t tsz = 0; 2487 int cnt = 0; 2488 mblk_t *mp; 2489 2490 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2491 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2492 if (((mac_srs->srs_bw->mac_bw_used + mac_srs->srs_size) <= 2493 mac_srs->srs_bw->mac_bw_limit) || 2494 (mac_srs->srs_bw->mac_bw_limit == 0)) { 2495 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2496 head = mac_srs->srs_first; 2497 mac_srs->srs_first = NULL; 2498 *chain_tail = mac_srs->srs_last; 2499 mac_srs->srs_last = NULL; 2500 *chain_sz = mac_srs->srs_size; 2501 *chain_cnt = mac_srs->srs_count; 2502 mac_srs->srs_count = 0; 2503 mac_srs->srs_size = 0; 2504 return (head); 2505 } 2506 2507 /* 2508 * Can't clear the entire backlog. 2509 * Need to find how many packets to pick 2510 */ 2511 ASSERT(MUTEX_HELD(&mac_srs->srs_bw->mac_bw_lock)); 2512 while ((mp = mac_srs->srs_first) != NULL) { 2513 sz = msgdsize(mp); 2514 if ((tsz + sz + mac_srs->srs_bw->mac_bw_used) > 2515 mac_srs->srs_bw->mac_bw_limit) { 2516 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) 2517 mac_srs->srs_bw->mac_bw_state |= 2518 SRS_BW_ENFORCED; 2519 break; 2520 } 2521 2522 /* 2523 * The _size & cnt is decremented from the softrings 2524 * when they send up the packet for polling to work 2525 * properly. 2526 */ 2527 tsz += sz; 2528 cnt++; 2529 mac_srs->srs_count--; 2530 mac_srs->srs_size -= sz; 2531 if (tail != NULL) 2532 tail->b_next = mp; 2533 else 2534 head = mp; 2535 tail = mp; 2536 mac_srs->srs_first = mac_srs->srs_first->b_next; 2537 } 2538 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2539 if (mac_srs->srs_first == NULL) 2540 mac_srs->srs_last = NULL; 2541 2542 if (tail != NULL) 2543 tail->b_next = NULL; 2544 *chain_tail = tail; 2545 *chain_cnt = cnt; 2546 *chain_sz = tsz; 2547 2548 return (head); 2549 } 2550 2551 /* 2552 * mac_rx_srs_drain 2553 * 2554 * The SRS drain routine. Gets to run to clear the queue. Any thread 2555 * (worker, interrupt, poll) can call this based on processing model. 2556 * The first thing we do is disable interrupts if possible and then 2557 * drain the queue. we also try to poll the underlying hardware if 2558 * there is a dedicated hardware Rx ring assigned to this SRS. 2559 * 2560 * There is a equivalent drain routine in bandwidth control mode 2561 * mac_rx_srs_drain_bw. There is some code duplication between the two 2562 * routines but they are highly performance sensitive and are easier 2563 * to read/debug if they stay separate. Any code changes here might 2564 * also apply to mac_rx_srs_drain_bw as well. 2565 */ 2566 void 2567 mac_rx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 2568 { 2569 mblk_t *head; 2570 mblk_t *tail; 2571 timeout_id_t tid; 2572 int cnt = 0; 2573 mac_client_impl_t *mcip = mac_srs->srs_mcip; 2574 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2575 2576 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2577 ASSERT(!(mac_srs->srs_type & SRST_BW_CONTROL)); 2578 2579 /* If we are blanked i.e. can't do upcalls, then we are done */ 2580 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) { 2581 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) || 2582 (mac_srs->srs_state & SRS_PAUSE)); 2583 goto out; 2584 } 2585 2586 if (mac_srs->srs_first == NULL) 2587 goto out; 2588 2589 if (!(mac_srs->srs_state & SRS_LATENCY_OPT) && 2590 (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)) { 2591 /* 2592 * In the normal case, the SRS worker thread does no 2593 * work and we wait for a backlog to build up before 2594 * we switch into polling mode. In case we are 2595 * optimizing for throughput, we use the worker thread 2596 * as well. The goal is to let worker thread process 2597 * the queue and poll thread to feed packets into 2598 * the queue. As such, we should signal the poll 2599 * thread to try and get more packets. 2600 * 2601 * We could have pulled this check in the POLL_RING 2602 * macro itself but keeping it explicit here makes 2603 * the architecture more human understandable. 2604 */ 2605 MAC_SRS_POLL_RING(mac_srs); 2606 } 2607 2608 again: 2609 head = mac_srs->srs_first; 2610 mac_srs->srs_first = NULL; 2611 tail = mac_srs->srs_last; 2612 mac_srs->srs_last = NULL; 2613 cnt = mac_srs->srs_count; 2614 mac_srs->srs_count = 0; 2615 2616 ASSERT(head != NULL); 2617 ASSERT(tail != NULL); 2618 2619 if ((tid = mac_srs->srs_tid) != NULL) 2620 mac_srs->srs_tid = NULL; 2621 2622 mac_srs->srs_state |= (SRS_PROC|proc_type); 2623 2624 2625 /* 2626 * mcip is NULL for broadcast and multicast flows. The promisc 2627 * callbacks for broadcast and multicast packets are delivered from 2628 * mac_rx() and we don't need to worry about that case in this path 2629 */ 2630 if (mcip != NULL) { 2631 if (mcip->mci_promisc_list != NULL) { 2632 mutex_exit(&mac_srs->srs_lock); 2633 mac_promisc_client_dispatch(mcip, head); 2634 mutex_enter(&mac_srs->srs_lock); 2635 } 2636 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) { 2637 mutex_exit(&mac_srs->srs_lock); 2638 mac_protect_intercept_dynamic(mcip, head); 2639 mutex_enter(&mac_srs->srs_lock); 2640 } 2641 } 2642 2643 /* 2644 * Check if SRS itself is doing the processing 2645 * This direct path does not apply when subflows are present. In this 2646 * case, packets need to be dispatched to a soft ring according to the 2647 * flow's bandwidth and other resources contraints. 2648 */ 2649 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) { 2650 mac_direct_rx_t proc; 2651 void *arg1; 2652 mac_resource_handle_t arg2; 2653 2654 /* 2655 * This is the case when a Rx is directly 2656 * assigned and we have a fully classified 2657 * protocol chain. We can deal with it in 2658 * one shot. 2659 */ 2660 proc = srs_rx->sr_func; 2661 arg1 = srs_rx->sr_arg1; 2662 arg2 = srs_rx->sr_arg2; 2663 2664 mac_srs->srs_state |= SRS_CLIENT_PROC; 2665 mutex_exit(&mac_srs->srs_lock); 2666 if (tid != NULL) { 2667 (void) untimeout(tid); 2668 tid = NULL; 2669 } 2670 2671 proc(arg1, arg2, head, NULL); 2672 /* 2673 * Decrement the size and count here itelf 2674 * since the packet has been processed. 2675 */ 2676 mutex_enter(&mac_srs->srs_lock); 2677 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 2678 if (mac_srs->srs_state & SRS_CLIENT_WAIT) 2679 cv_signal(&mac_srs->srs_client_cv); 2680 mac_srs->srs_state &= ~SRS_CLIENT_PROC; 2681 } else { 2682 /* Some kind of softrings based fanout is required */ 2683 mutex_exit(&mac_srs->srs_lock); 2684 if (tid != NULL) { 2685 (void) untimeout(tid); 2686 tid = NULL; 2687 } 2688 2689 /* 2690 * Since the fanout routines can deal with chains, 2691 * shoot the entire chain up. 2692 */ 2693 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP) 2694 mac_rx_srs_fanout(mac_srs, head); 2695 else 2696 mac_rx_srs_proto_fanout(mac_srs, head); 2697 mutex_enter(&mac_srs->srs_lock); 2698 } 2699 2700 if (!(mac_srs->srs_state & (SRS_BLANK|SRS_PAUSE)) && 2701 (mac_srs->srs_first != NULL)) { 2702 /* 2703 * More packets arrived while we were clearing the 2704 * SRS. This can be possible because of one of 2705 * three conditions below: 2706 * 1) The driver is using multiple worker threads 2707 * to send the packets to us. 2708 * 2) The driver has a race in switching 2709 * between interrupt and polling mode or 2710 * 3) Packets are arriving in this SRS via the 2711 * S/W classification as well. 2712 * 2713 * We should switch to polling mode and see if we 2714 * need to send the poll thread down. Also, signal 2715 * the worker thread to process whats just arrived. 2716 */ 2717 MAC_SRS_POLLING_ON(mac_srs); 2718 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) { 2719 srs_rx->sr_drain_poll_sig++; 2720 MAC_SRS_POLL_RING(mac_srs); 2721 } 2722 2723 /* 2724 * If we didn't signal the poll thread, we need 2725 * to deal with the pending packets ourselves. 2726 */ 2727 if (proc_type == SRS_WORKER) { 2728 srs_rx->sr_drain_again++; 2729 goto again; 2730 } else { 2731 srs_rx->sr_drain_worker_sig++; 2732 cv_signal(&mac_srs->srs_async); 2733 } 2734 } 2735 2736 out: 2737 if (mac_srs->srs_state & SRS_GET_PKTS) { 2738 /* 2739 * Poll thread is already running. Leave the 2740 * SRS_RPOC set and hand over the control to 2741 * poll thread. 2742 */ 2743 mac_srs->srs_state &= ~proc_type; 2744 srs_rx->sr_drain_poll_running++; 2745 return; 2746 } 2747 2748 /* 2749 * Even if there are no packets queued in SRS, we 2750 * need to make sure that the shared counter is 2751 * clear and any associated softrings have cleared 2752 * all the backlog. Otherwise, leave the interface 2753 * in polling mode and the poll thread will get 2754 * signalled once the count goes down to zero. 2755 * 2756 * If someone is already draining the queue (SRS_PROC is 2757 * set) when the srs_poll_pkt_cnt goes down to zero, 2758 * then it means that drain is already running and we 2759 * will turn off polling at that time if there is 2760 * no backlog. 2761 * 2762 * As long as there are packets queued either 2763 * in soft ring set or its soft rings, we will leave 2764 * the interface in polling mode (even if the drain 2765 * was done being the interrupt thread). We signal 2766 * the poll thread as well if we have dipped below 2767 * low water mark. 2768 * 2769 * NOTE: We can't use the MAC_SRS_POLLING_ON macro 2770 * since that turn polling on only for worker thread. 2771 * Its not worth turning polling on for interrupt 2772 * thread (since NIC will not issue another interrupt) 2773 * unless a backlog builds up. 2774 */ 2775 if ((srs_rx->sr_poll_pkt_cnt > 0) && 2776 (mac_srs->srs_state & SRS_POLLING_CAPAB)) { 2777 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 2778 srs_rx->sr_drain_keep_polling++; 2779 MAC_SRS_POLLING_ON(mac_srs); 2780 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) 2781 MAC_SRS_POLL_RING(mac_srs); 2782 return; 2783 } 2784 2785 /* Nothing else to do. Get out of poll mode */ 2786 MAC_SRS_POLLING_OFF(mac_srs); 2787 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 2788 srs_rx->sr_drain_finish_intr++; 2789 } 2790 2791 /* 2792 * mac_rx_srs_drain_bw 2793 * 2794 * The SRS BW drain routine. Gets to run to clear the queue. Any thread 2795 * (worker, interrupt, poll) can call this based on processing model. 2796 * The first thing we do is disable interrupts if possible and then 2797 * drain the queue. we also try to poll the underlying hardware if 2798 * there is a dedicated hardware Rx ring assigned to this SRS. 2799 * 2800 * There is a equivalent drain routine in non bandwidth control mode 2801 * mac_rx_srs_drain. There is some code duplication between the two 2802 * routines but they are highly performance sensitive and are easier 2803 * to read/debug if they stay separate. Any code changes here might 2804 * also apply to mac_rx_srs_drain as well. 2805 */ 2806 void 2807 mac_rx_srs_drain_bw(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 2808 { 2809 mblk_t *head; 2810 mblk_t *tail; 2811 timeout_id_t tid; 2812 size_t sz = 0; 2813 int cnt = 0; 2814 mac_client_impl_t *mcip = mac_srs->srs_mcip; 2815 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 2816 clock_t now; 2817 2818 ASSERT(MUTEX_HELD(&mac_srs->srs_lock)); 2819 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL); 2820 again: 2821 /* Check if we are doing B/W control */ 2822 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2823 now = ddi_get_lbolt(); 2824 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 2825 mac_srs->srs_bw->mac_bw_curr_time = now; 2826 mac_srs->srs_bw->mac_bw_used = 0; 2827 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) 2828 mac_srs->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; 2829 } else if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) { 2830 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2831 goto done; 2832 } else if (mac_srs->srs_bw->mac_bw_used > 2833 mac_srs->srs_bw->mac_bw_limit) { 2834 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 2835 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2836 goto done; 2837 } 2838 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2839 2840 /* If we are blanked i.e. can't do upcalls, then we are done */ 2841 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) { 2842 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) || 2843 (mac_srs->srs_state & SRS_PAUSE)); 2844 goto done; 2845 } 2846 2847 sz = 0; 2848 cnt = 0; 2849 if ((head = mac_srs_pick_chain(mac_srs, &tail, &sz, &cnt)) == NULL) { 2850 /* 2851 * We couldn't pick up a single packet. 2852 */ 2853 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2854 if ((mac_srs->srs_bw->mac_bw_used == 0) && 2855 (mac_srs->srs_size != 0) && 2856 !(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 2857 /* 2858 * Seems like configured B/W doesn't 2859 * even allow processing of 1 packet 2860 * per tick. 2861 * 2862 * XXX: raise the limit to processing 2863 * at least 1 packet per tick. 2864 */ 2865 mac_srs->srs_bw->mac_bw_limit += 2866 mac_srs->srs_bw->mac_bw_limit; 2867 mac_srs->srs_bw->mac_bw_drop_threshold += 2868 mac_srs->srs_bw->mac_bw_drop_threshold; 2869 cmn_err(CE_NOTE, "mac_rx_srs_drain: srs(%p) " 2870 "raised B/W limit to %d since not even a " 2871 "single packet can be processed per " 2872 "tick %d\n", (void *)mac_srs, 2873 (int)mac_srs->srs_bw->mac_bw_limit, 2874 (int)msgdsize(mac_srs->srs_first)); 2875 } 2876 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2877 goto done; 2878 } 2879 2880 ASSERT(head != NULL); 2881 ASSERT(tail != NULL); 2882 2883 /* zero bandwidth: drop all and return to interrupt mode */ 2884 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2885 if (mac_srs->srs_bw->mac_bw_limit == 0) { 2886 srs_rx->sr_stat.mrs_sdrops += cnt; 2887 ASSERT(mac_srs->srs_bw->mac_bw_sz >= sz); 2888 mac_srs->srs_bw->mac_bw_sz -= sz; 2889 mac_srs->srs_bw->mac_bw_drop_bytes += sz; 2890 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2891 mac_pkt_drop(NULL, NULL, head, B_FALSE); 2892 goto leave_poll; 2893 } else { 2894 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 2895 } 2896 2897 if ((tid = mac_srs->srs_tid) != NULL) 2898 mac_srs->srs_tid = NULL; 2899 2900 mac_srs->srs_state |= (SRS_PROC|proc_type); 2901 MAC_SRS_WORKER_POLLING_ON(mac_srs); 2902 2903 /* 2904 * mcip is NULL for broadcast and multicast flows. The promisc 2905 * callbacks for broadcast and multicast packets are delivered from 2906 * mac_rx() and we don't need to worry about that case in this path 2907 */ 2908 if (mcip != NULL) { 2909 if (mcip->mci_promisc_list != NULL) { 2910 mutex_exit(&mac_srs->srs_lock); 2911 mac_promisc_client_dispatch(mcip, head); 2912 mutex_enter(&mac_srs->srs_lock); 2913 } 2914 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) { 2915 mutex_exit(&mac_srs->srs_lock); 2916 mac_protect_intercept_dynamic(mcip, head); 2917 mutex_enter(&mac_srs->srs_lock); 2918 } 2919 } 2920 2921 /* 2922 * Check if SRS itself is doing the processing 2923 * This direct path does not apply when subflows are present. In this 2924 * case, packets need to be dispatched to a soft ring according to the 2925 * flow's bandwidth and other resources contraints. 2926 */ 2927 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) { 2928 mac_direct_rx_t proc; 2929 void *arg1; 2930 mac_resource_handle_t arg2; 2931 2932 /* 2933 * This is the case when a Rx is directly 2934 * assigned and we have a fully classified 2935 * protocol chain. We can deal with it in 2936 * one shot. 2937 */ 2938 proc = srs_rx->sr_func; 2939 arg1 = srs_rx->sr_arg1; 2940 arg2 = srs_rx->sr_arg2; 2941 2942 mac_srs->srs_state |= SRS_CLIENT_PROC; 2943 mutex_exit(&mac_srs->srs_lock); 2944 if (tid != NULL) { 2945 (void) untimeout(tid); 2946 tid = NULL; 2947 } 2948 2949 proc(arg1, arg2, head, NULL); 2950 /* 2951 * Decrement the size and count here itelf 2952 * since the packet has been processed. 2953 */ 2954 mutex_enter(&mac_srs->srs_lock); 2955 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 2956 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz); 2957 2958 if (mac_srs->srs_state & SRS_CLIENT_WAIT) 2959 cv_signal(&mac_srs->srs_client_cv); 2960 mac_srs->srs_state &= ~SRS_CLIENT_PROC; 2961 } else { 2962 /* Some kind of softrings based fanout is required */ 2963 mutex_exit(&mac_srs->srs_lock); 2964 if (tid != NULL) { 2965 (void) untimeout(tid); 2966 tid = NULL; 2967 } 2968 2969 /* 2970 * Since the fanout routines can deal with chains, 2971 * shoot the entire chain up. 2972 */ 2973 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP) 2974 mac_rx_srs_fanout(mac_srs, head); 2975 else 2976 mac_rx_srs_proto_fanout(mac_srs, head); 2977 mutex_enter(&mac_srs->srs_lock); 2978 } 2979 2980 /* 2981 * Send the poll thread to pick up any packets arrived 2982 * so far. This also serves as the last check in case 2983 * nothing else is queued in the SRS. The poll thread 2984 * is signalled only in the case the drain was done 2985 * by the worker thread and SRS_WORKER is set. The 2986 * worker thread can run in parallel as long as the 2987 * SRS_WORKER flag is set. We we have nothing else to 2988 * process, we can exit while leaving SRS_PROC set 2989 * which gives the poll thread control to process and 2990 * cleanup once it returns from the NIC. 2991 * 2992 * If we have nothing else to process, we need to 2993 * ensure that we keep holding the srs_lock till 2994 * all the checks below are done and control is 2995 * handed to the poll thread if it was running. 2996 */ 2997 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 2998 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 2999 if (mac_srs->srs_first != NULL) { 3000 if (proc_type == SRS_WORKER) { 3001 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3002 if (srs_rx->sr_poll_pkt_cnt <= 3003 srs_rx->sr_lowat) 3004 MAC_SRS_POLL_RING(mac_srs); 3005 goto again; 3006 } else { 3007 cv_signal(&mac_srs->srs_async); 3008 } 3009 } 3010 } 3011 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3012 3013 done: 3014 3015 if (mac_srs->srs_state & SRS_GET_PKTS) { 3016 /* 3017 * Poll thread is already running. Leave the 3018 * SRS_RPOC set and hand over the control to 3019 * poll thread. 3020 */ 3021 mac_srs->srs_state &= ~proc_type; 3022 return; 3023 } 3024 3025 /* 3026 * If we can't process packets because we have exceeded 3027 * B/W limit for this tick, just set the timeout 3028 * and leave. 3029 * 3030 * Even if there are no packets queued in SRS, we 3031 * need to make sure that the shared counter is 3032 * clear and any associated softrings have cleared 3033 * all the backlog. Otherwise, leave the interface 3034 * in polling mode and the poll thread will get 3035 * signalled once the count goes down to zero. 3036 * 3037 * If someone is already draining the queue (SRS_PROC is 3038 * set) when the srs_poll_pkt_cnt goes down to zero, 3039 * then it means that drain is already running and we 3040 * will turn off polling at that time if there is 3041 * no backlog. As long as there are packets queued either 3042 * is soft ring set or its soft rings, we will leave 3043 * the interface in polling mode. 3044 */ 3045 mutex_enter(&mac_srs->srs_bw->mac_bw_lock); 3046 if ((mac_srs->srs_state & SRS_POLLING_CAPAB) && 3047 ((mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) || 3048 (srs_rx->sr_poll_pkt_cnt > 0))) { 3049 MAC_SRS_POLLING_ON(mac_srs); 3050 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 3051 if ((mac_srs->srs_first != NULL) && 3052 (mac_srs->srs_tid == NULL)) 3053 mac_srs->srs_tid = timeout(mac_srs_fire, 3054 mac_srs, 1); 3055 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3056 return; 3057 } 3058 mutex_exit(&mac_srs->srs_bw->mac_bw_lock); 3059 3060 leave_poll: 3061 3062 /* Nothing else to do. Get out of poll mode */ 3063 MAC_SRS_POLLING_OFF(mac_srs); 3064 mac_srs->srs_state &= ~(SRS_PROC|proc_type); 3065 } 3066 3067 /* 3068 * mac_srs_worker 3069 * 3070 * The SRS worker routine. Drains the queue when no one else is 3071 * processing it. 3072 */ 3073 void 3074 mac_srs_worker(mac_soft_ring_set_t *mac_srs) 3075 { 3076 kmutex_t *lock = &mac_srs->srs_lock; 3077 kcondvar_t *async = &mac_srs->srs_async; 3078 callb_cpr_t cprinfo; 3079 boolean_t bw_ctl_flag; 3080 3081 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "srs_worker"); 3082 mutex_enter(lock); 3083 3084 start: 3085 for (;;) { 3086 bw_ctl_flag = B_FALSE; 3087 if (mac_srs->srs_type & SRST_BW_CONTROL) { 3088 MAC_SRS_BW_LOCK(mac_srs); 3089 MAC_SRS_CHECK_BW_CONTROL(mac_srs); 3090 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) 3091 bw_ctl_flag = B_TRUE; 3092 MAC_SRS_BW_UNLOCK(mac_srs); 3093 } 3094 /* 3095 * The SRS_BW_ENFORCED flag may change since we have dropped 3096 * the mac_bw_lock. However the drain function can handle both 3097 * a drainable SRS or a bandwidth controlled SRS, and the 3098 * effect of scheduling a timeout is to wakeup the worker 3099 * thread which in turn will call the drain function. Since 3100 * we release the srs_lock atomically only in the cv_wait there 3101 * isn't a fear of waiting for ever. 3102 */ 3103 while (((mac_srs->srs_state & SRS_PROC) || 3104 (mac_srs->srs_first == NULL) || bw_ctl_flag || 3105 (mac_srs->srs_state & SRS_TX_BLOCKED)) && 3106 !(mac_srs->srs_state & SRS_PAUSE)) { 3107 /* 3108 * If we have packets queued and we are here 3109 * because B/W control is in place, we better 3110 * schedule the worker wakeup after 1 tick 3111 * to see if bandwidth control can be relaxed. 3112 */ 3113 if (bw_ctl_flag && mac_srs->srs_tid == NULL) { 3114 /* 3115 * We need to ensure that a timer is already 3116 * scheduled or we force schedule one for 3117 * later so that we can continue processing 3118 * after this quanta is over. 3119 */ 3120 mac_srs->srs_tid = timeout(mac_srs_fire, 3121 mac_srs, 1); 3122 } 3123 wait: 3124 CALLB_CPR_SAFE_BEGIN(&cprinfo); 3125 cv_wait(async, lock); 3126 CALLB_CPR_SAFE_END(&cprinfo, lock); 3127 3128 if (mac_srs->srs_state & SRS_PAUSE) 3129 goto done; 3130 if (mac_srs->srs_state & SRS_PROC) 3131 goto wait; 3132 3133 if (mac_srs->srs_first != NULL && 3134 mac_srs->srs_type & SRST_BW_CONTROL) { 3135 MAC_SRS_BW_LOCK(mac_srs); 3136 if (mac_srs->srs_bw->mac_bw_state & 3137 SRS_BW_ENFORCED) { 3138 MAC_SRS_CHECK_BW_CONTROL(mac_srs); 3139 } 3140 bw_ctl_flag = mac_srs->srs_bw->mac_bw_state & 3141 SRS_BW_ENFORCED; 3142 MAC_SRS_BW_UNLOCK(mac_srs); 3143 } 3144 } 3145 3146 if (mac_srs->srs_state & SRS_PAUSE) 3147 goto done; 3148 mac_srs->srs_drain_func(mac_srs, SRS_WORKER); 3149 } 3150 done: 3151 /* 3152 * The Rx SRS quiesce logic first cuts off packet supply to the SRS 3153 * from both hard and soft classifications and waits for such threads 3154 * to finish before signaling the worker. So at this point the only 3155 * thread left that could be competing with the worker is the poll 3156 * thread. In the case of Tx, there shouldn't be any thread holding 3157 * SRS_PROC at this point. 3158 */ 3159 if (!(mac_srs->srs_state & SRS_PROC)) { 3160 mac_srs->srs_state |= SRS_PROC; 3161 } else { 3162 ASSERT((mac_srs->srs_type & SRST_TX) == 0); 3163 /* 3164 * Poll thread still owns the SRS and is still running 3165 */ 3166 ASSERT((mac_srs->srs_poll_thr == NULL) || 3167 ((mac_srs->srs_state & SRS_POLL_THR_OWNER) == 3168 SRS_POLL_THR_OWNER)); 3169 } 3170 mac_srs_worker_quiesce(mac_srs); 3171 /* 3172 * Wait for the SRS_RESTART or SRS_CONDEMNED signal from the initiator 3173 * of the quiesce operation 3174 */ 3175 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_RESTART))) 3176 cv_wait(&mac_srs->srs_async, &mac_srs->srs_lock); 3177 3178 if (mac_srs->srs_state & SRS_RESTART) { 3179 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED)); 3180 mac_srs_worker_restart(mac_srs); 3181 mac_srs->srs_state &= ~SRS_PROC; 3182 goto start; 3183 } 3184 3185 if (!(mac_srs->srs_state & SRS_CONDEMNED_DONE)) 3186 mac_srs_worker_quiesce(mac_srs); 3187 3188 mac_srs->srs_state &= ~SRS_PROC; 3189 /* The macro drops the srs_lock */ 3190 CALLB_CPR_EXIT(&cprinfo); 3191 thread_exit(); 3192 } 3193 3194 /* 3195 * mac_rx_srs_subflow_process 3196 * 3197 * Receive side routine called from interrupt path when there are 3198 * sub flows present on this SRS. 3199 */ 3200 /* ARGSUSED */ 3201 void 3202 mac_rx_srs_subflow_process(void *arg, mac_resource_handle_t srs, 3203 mblk_t *mp_chain, boolean_t loopback) 3204 { 3205 flow_entry_t *flent = NULL; 3206 flow_entry_t *prev_flent = NULL; 3207 mblk_t *mp = NULL; 3208 mblk_t *tail = NULL; 3209 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs; 3210 mac_client_impl_t *mcip; 3211 3212 mcip = mac_srs->srs_mcip; 3213 ASSERT(mcip != NULL); 3214 3215 /* 3216 * We need to determine the SRS for every packet 3217 * by walking the flow table, if we don't get any, 3218 * then we proceed using the SRS we came with. 3219 */ 3220 mp = tail = mp_chain; 3221 while (mp != NULL) { 3222 3223 /* 3224 * We will increment the stats for the mactching subflow. 3225 * when we get the bytes/pkt count for the classified packets 3226 * later in mac_rx_srs_process. 3227 */ 3228 (void) mac_flow_lookup(mcip->mci_subflow_tab, mp, 3229 FLOW_INBOUND, &flent); 3230 3231 if (mp == mp_chain || flent == prev_flent) { 3232 if (prev_flent != NULL) 3233 FLOW_REFRELE(prev_flent); 3234 prev_flent = flent; 3235 flent = NULL; 3236 tail = mp; 3237 mp = mp->b_next; 3238 continue; 3239 } 3240 tail->b_next = NULL; 3241 /* 3242 * A null indicates, this is for the mac_srs itself. 3243 * XXX-venu : probably assert for fe_rx_srs_cnt == 0. 3244 */ 3245 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) { 3246 mac_rx_srs_process(arg, 3247 (mac_resource_handle_t)mac_srs, mp_chain, 3248 loopback); 3249 } else { 3250 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1, 3251 prev_flent->fe_cb_arg2, mp_chain, loopback); 3252 FLOW_REFRELE(prev_flent); 3253 } 3254 prev_flent = flent; 3255 flent = NULL; 3256 mp_chain = mp; 3257 tail = mp; 3258 mp = mp->b_next; 3259 } 3260 /* Last chain */ 3261 ASSERT(mp_chain != NULL); 3262 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) { 3263 mac_rx_srs_process(arg, 3264 (mac_resource_handle_t)mac_srs, mp_chain, loopback); 3265 } else { 3266 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1, 3267 prev_flent->fe_cb_arg2, mp_chain, loopback); 3268 FLOW_REFRELE(prev_flent); 3269 } 3270 } 3271 3272 /* 3273 * mac_rx_srs_process 3274 * 3275 * Receive side routine called from the interrupt path. 3276 * 3277 * loopback is set to force a context switch on the loopback 3278 * path between MAC clients. 3279 */ 3280 /* ARGSUSED */ 3281 void 3282 mac_rx_srs_process(void *arg, mac_resource_handle_t srs, mblk_t *mp_chain, 3283 boolean_t loopback) 3284 { 3285 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs; 3286 mblk_t *mp, *tail, *head; 3287 int count = 0; 3288 int count1; 3289 size_t sz = 0; 3290 size_t chain_sz, sz1; 3291 mac_bw_ctl_t *mac_bw; 3292 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx; 3293 3294 /* 3295 * Set the tail, count and sz. We set the sz irrespective 3296 * of whether we are doing B/W control or not for the 3297 * purpose of updating the stats. 3298 */ 3299 mp = tail = mp_chain; 3300 while (mp != NULL) { 3301 tail = mp; 3302 count++; 3303 sz += msgdsize(mp); 3304 mp = mp->b_next; 3305 } 3306 3307 mutex_enter(&mac_srs->srs_lock); 3308 3309 if (loopback) { 3310 SRS_RX_STAT_UPDATE(mac_srs, lclbytes, sz); 3311 SRS_RX_STAT_UPDATE(mac_srs, lclcnt, count); 3312 3313 } else { 3314 SRS_RX_STAT_UPDATE(mac_srs, intrbytes, sz); 3315 SRS_RX_STAT_UPDATE(mac_srs, intrcnt, count); 3316 } 3317 3318 /* 3319 * If the SRS in already being processed; has been blanked; 3320 * can be processed by worker thread only; or the B/W limit 3321 * has been reached, then queue the chain and check if 3322 * worker thread needs to be awakend. 3323 */ 3324 if (mac_srs->srs_type & SRST_BW_CONTROL) { 3325 mac_bw = mac_srs->srs_bw; 3326 ASSERT(mac_bw != NULL); 3327 mutex_enter(&mac_bw->mac_bw_lock); 3328 mac_bw->mac_bw_intr += sz; 3329 if (mac_bw->mac_bw_limit == 0) { 3330 /* zero bandwidth: drop all */ 3331 srs_rx->sr_stat.mrs_sdrops += count; 3332 mac_bw->mac_bw_drop_bytes += sz; 3333 mutex_exit(&mac_bw->mac_bw_lock); 3334 mutex_exit(&mac_srs->srs_lock); 3335 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE); 3336 return; 3337 } else { 3338 if ((mac_bw->mac_bw_sz + sz) <= 3339 mac_bw->mac_bw_drop_threshold) { 3340 mutex_exit(&mac_bw->mac_bw_lock); 3341 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, 3342 tail, count, sz); 3343 } else { 3344 mp = mp_chain; 3345 chain_sz = 0; 3346 count1 = 0; 3347 tail = NULL; 3348 head = NULL; 3349 while (mp != NULL) { 3350 sz1 = msgdsize(mp); 3351 if (mac_bw->mac_bw_sz + chain_sz + sz1 > 3352 mac_bw->mac_bw_drop_threshold) 3353 break; 3354 chain_sz += sz1; 3355 count1++; 3356 tail = mp; 3357 mp = mp->b_next; 3358 } 3359 mutex_exit(&mac_bw->mac_bw_lock); 3360 if (tail != NULL) { 3361 head = tail->b_next; 3362 tail->b_next = NULL; 3363 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, 3364 mp_chain, tail, count1, chain_sz); 3365 sz -= chain_sz; 3366 count -= count1; 3367 } else { 3368 /* Can't pick up any */ 3369 head = mp_chain; 3370 } 3371 if (head != NULL) { 3372 /* Drop any packet over the threshold */ 3373 srs_rx->sr_stat.mrs_sdrops += count; 3374 mutex_enter(&mac_bw->mac_bw_lock); 3375 mac_bw->mac_bw_drop_bytes += sz; 3376 mutex_exit(&mac_bw->mac_bw_lock); 3377 freemsgchain(head); 3378 } 3379 } 3380 MAC_SRS_WORKER_WAKEUP(mac_srs); 3381 mutex_exit(&mac_srs->srs_lock); 3382 return; 3383 } 3384 } 3385 3386 /* 3387 * If the total number of packets queued in the SRS and 3388 * its associated soft rings exceeds the max allowed, 3389 * then drop the chain. If we are polling capable, this 3390 * shouldn't be happening. 3391 */ 3392 if (!(mac_srs->srs_type & SRST_BW_CONTROL) && 3393 (srs_rx->sr_poll_pkt_cnt > srs_rx->sr_hiwat)) { 3394 mac_bw = mac_srs->srs_bw; 3395 srs_rx->sr_stat.mrs_sdrops += count; 3396 mutex_enter(&mac_bw->mac_bw_lock); 3397 mac_bw->mac_bw_drop_bytes += sz; 3398 mutex_exit(&mac_bw->mac_bw_lock); 3399 freemsgchain(mp_chain); 3400 mutex_exit(&mac_srs->srs_lock); 3401 return; 3402 } 3403 3404 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, tail, count, sz); 3405 3406 if (!(mac_srs->srs_state & SRS_PROC)) { 3407 /* 3408 * If we are coming via loopback, if we are not optimizing for 3409 * latency, or if our stack is running deep, we should signal 3410 * the worker thread. 3411 */ 3412 if (loopback || !(mac_srs->srs_state & SRS_LATENCY_OPT) || 3413 MAC_RX_SRS_TOODEEP()) { 3414 /* 3415 * For loopback, We need to let the worker take 3416 * over as we don't want to continue in the same 3417 * thread even if we can. This could lead to stack 3418 * overflows and may also end up using 3419 * resources (cpu) incorrectly. 3420 */ 3421 cv_signal(&mac_srs->srs_async); 3422 } else { 3423 /* 3424 * Seems like no one is processing the SRS and 3425 * there is no backlog. We also inline process 3426 * our packet if its a single packet in non 3427 * latency optimized case (in latency optimized 3428 * case, we inline process chains of any size). 3429 */ 3430 mac_srs->srs_drain_func(mac_srs, SRS_PROC_FAST); 3431 } 3432 } 3433 mutex_exit(&mac_srs->srs_lock); 3434 } 3435 3436 /* TX SIDE ROUTINES (RUNTIME) */ 3437 3438 /* 3439 * mac_tx_srs_no_desc 3440 * 3441 * This routine is called by Tx single ring default mode 3442 * when Tx ring runs out of descs. 3443 */ 3444 mac_tx_cookie_t 3445 mac_tx_srs_no_desc(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3446 uint16_t flag, mblk_t **ret_mp) 3447 { 3448 mac_tx_cookie_t cookie = 0; 3449 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3450 boolean_t wakeup_worker = B_TRUE; 3451 uint32_t tx_mode = srs_tx->st_mode; 3452 int cnt, sz; 3453 mblk_t *tail; 3454 3455 ASSERT(tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_BW); 3456 if (flag & MAC_DROP_ON_NO_DESC) { 3457 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie); 3458 } else { 3459 if (mac_srs->srs_first != NULL) 3460 wakeup_worker = B_FALSE; 3461 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3462 if (flag & MAC_TX_NO_ENQUEUE) { 3463 /* 3464 * If TX_QUEUED is not set, queue the 3465 * packet and let mac_tx_srs_drain() 3466 * set the TX_BLOCKED bit for the 3467 * reasons explained above. Otherwise, 3468 * return the mblks. 3469 */ 3470 if (wakeup_worker) { 3471 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3472 mp_chain, tail, cnt, sz); 3473 } else { 3474 MAC_TX_SET_NO_ENQUEUE(mac_srs, 3475 mp_chain, ret_mp, cookie); 3476 } 3477 } else { 3478 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain, 3479 tail, cnt, sz, cookie); 3480 } 3481 if (wakeup_worker) 3482 cv_signal(&mac_srs->srs_async); 3483 } 3484 return (cookie); 3485 } 3486 3487 /* 3488 * mac_tx_srs_enqueue 3489 * 3490 * This routine is called when Tx SRS is operating in either serializer 3491 * or bandwidth mode. In serializer mode, a packet will get enqueued 3492 * when a thread cannot enter SRS exclusively. In bandwidth mode, 3493 * packets gets queued if allowed byte-count limit for a tick is 3494 * exceeded. The action that gets taken when MAC_DROP_ON_NO_DESC and 3495 * MAC_TX_NO_ENQUEUE is set is different than when operaing in either 3496 * the default mode or fanout mode. Here packets get dropped or 3497 * returned back to the caller only after hi-watermark worth of data 3498 * is queued. 3499 */ 3500 static mac_tx_cookie_t 3501 mac_tx_srs_enqueue(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3502 uint16_t flag, uintptr_t fanout_hint, mblk_t **ret_mp) 3503 { 3504 mac_tx_cookie_t cookie = 0; 3505 int cnt, sz; 3506 mblk_t *tail; 3507 boolean_t wakeup_worker = B_TRUE; 3508 3509 /* 3510 * Ignore fanout hint if we don't have multiple tx rings. 3511 */ 3512 if (!MAC_TX_SOFT_RINGS(mac_srs)) 3513 fanout_hint = 0; 3514 3515 if (mac_srs->srs_first != NULL) 3516 wakeup_worker = B_FALSE; 3517 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3518 if (flag & MAC_DROP_ON_NO_DESC) { 3519 if (mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) { 3520 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie); 3521 } else { 3522 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3523 mp_chain, tail, cnt, sz); 3524 } 3525 } else if (flag & MAC_TX_NO_ENQUEUE) { 3526 if ((mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) || 3527 (mac_srs->srs_state & SRS_TX_WAKEUP_CLIENT)) { 3528 MAC_TX_SET_NO_ENQUEUE(mac_srs, mp_chain, 3529 ret_mp, cookie); 3530 } else { 3531 mp_chain->b_prev = (mblk_t *)fanout_hint; 3532 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3533 mp_chain, tail, cnt, sz); 3534 } 3535 } else { 3536 /* 3537 * If you are BW_ENFORCED, just enqueue the 3538 * packet. srs_worker will drain it at the 3539 * prescribed rate. Before enqueueing, save 3540 * the fanout hint. 3541 */ 3542 mp_chain->b_prev = (mblk_t *)fanout_hint; 3543 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain, 3544 tail, cnt, sz, cookie); 3545 } 3546 if (wakeup_worker) 3547 cv_signal(&mac_srs->srs_async); 3548 return (cookie); 3549 } 3550 3551 /* 3552 * There are seven tx modes: 3553 * 3554 * 1) Default mode (SRS_TX_DEFAULT) 3555 * 2) Serialization mode (SRS_TX_SERIALIZE) 3556 * 3) Fanout mode (SRS_TX_FANOUT) 3557 * 4) Bandwdith mode (SRS_TX_BW) 3558 * 5) Fanout and Bandwidth mode (SRS_TX_BW_FANOUT) 3559 * 6) aggr Tx mode (SRS_TX_AGGR) 3560 * 7) aggr Tx bw mode (SRS_TX_BW_AGGR) 3561 * 3562 * The tx mode in which an SRS operates is decided in mac_tx_srs_setup() 3563 * based on the number of Tx rings requested for an SRS and whether 3564 * bandwidth control is requested or not. 3565 * 3566 * The default mode (i.e., no fanout/no bandwidth) is used when the 3567 * underlying NIC does not have Tx rings or just one Tx ring. In this mode, 3568 * the SRS acts as a pass-thru. Packets will go directly to mac_tx_send(). 3569 * When the underlying Tx ring runs out of Tx descs, it starts queueing up 3570 * packets in SRS. When flow-control is relieved, the srs_worker drains 3571 * the queued packets and informs blocked clients to restart sending 3572 * packets. 3573 * 3574 * In the SRS_TX_SERIALIZE mode, all calls to mac_tx() are serialized. This 3575 * mode is used when the link has no Tx rings or only one Tx ring. 3576 * 3577 * In the SRS_TX_FANOUT mode, packets will be fanned out to multiple 3578 * Tx rings. Each Tx ring will have a soft ring associated with it. 3579 * These soft rings will be hung off the Tx SRS. Queueing if it happens 3580 * due to lack of Tx desc will be in individual soft ring (and not srs) 3581 * associated with Tx ring. 3582 * 3583 * In the TX_BW mode, tx srs will allow packets to go down to Tx ring 3584 * only if bw is available. Otherwise the packets will be queued in 3585 * SRS. If fanout to multiple Tx rings is configured, the packets will 3586 * be fanned out among the soft rings associated with the Tx rings. 3587 * 3588 * In SRS_TX_AGGR mode, mac_tx_aggr_mode() routine is called. This routine 3589 * invokes an aggr function, aggr_find_tx_ring(), to find a pseudo Tx ring 3590 * belonging to a port on which the packet has to be sent. Aggr will 3591 * always have a pseudo Tx ring associated with it even when it is an 3592 * aggregation over a single NIC that has no Tx rings. Even in such a 3593 * case, the single pseudo Tx ring will have a soft ring associated with 3594 * it and the soft ring will hang off the SRS. 3595 * 3596 * If a bandwidth is specified for an aggr, SRS_TX_BW_AGGR mode is used. 3597 * In this mode, the bandwidth is first applied on the outgoing packets 3598 * and later mac_tx_addr_mode() function is called to send the packet out 3599 * of one of the pseudo Tx rings. 3600 * 3601 * Four flags are used in srs_state for indicating flow control 3602 * conditions : SRS_TX_BLOCKED, SRS_TX_HIWAT, SRS_TX_WAKEUP_CLIENT. 3603 * SRS_TX_BLOCKED indicates out of Tx descs. SRS expects a wakeup from the 3604 * driver below. 3605 * SRS_TX_HIWAT indicates packet count enqueued in Tx SRS exceeded Tx hiwat 3606 * and flow-control pressure is applied back to clients. The clients expect 3607 * wakeup when flow-control is relieved. 3608 * SRS_TX_WAKEUP_CLIENT get set when (flag == MAC_TX_NO_ENQUEUE) and mblk 3609 * got returned back to client either due to lack of Tx descs or due to bw 3610 * control reasons. The clients expect a wakeup when condition is relieved. 3611 * 3612 * The fourth argument to mac_tx() is the flag. Normally it will be 0 but 3613 * some clients set the following values too: MAC_DROP_ON_NO_DESC, 3614 * MAC_TX_NO_ENQUEUE 3615 * Mac clients that do not want packets to be enqueued in the mac layer set 3616 * MAC_DROP_ON_NO_DESC value. The packets won't be queued in the Tx SRS or 3617 * Tx soft rings but instead get dropped when the NIC runs out of desc. The 3618 * behaviour of this flag is different when the Tx is running in serializer 3619 * or bandwidth mode. Under these (Serializer, bandwidth) modes, the packet 3620 * get dropped when Tx high watermark is reached. 3621 * There are some mac clients like vsw, aggr that want the mblks to be 3622 * returned back to clients instead of being queued in Tx SRS (or Tx soft 3623 * rings) under flow-control (i.e., out of desc or exceeding bw limits) 3624 * conditions. These clients call mac_tx() with MAC_TX_NO_ENQUEUE flag set. 3625 * In the default and Tx fanout mode, the un-transmitted mblks will be 3626 * returned back to the clients when the driver runs out of Tx descs. 3627 * SRS_TX_WAKEUP_CLIENT (or S_RING_WAKEUP_CLIENT) will be set in SRS (or 3628 * soft ring) so that the clients can be woken up when Tx desc become 3629 * available. When running in serializer or bandwidth mode mode, 3630 * SRS_TX_WAKEUP_CLIENT will be set when tx hi-watermark is reached. 3631 */ 3632 3633 mac_tx_func_t 3634 mac_tx_get_func(uint32_t mode) 3635 { 3636 return (mac_tx_mode_list[mode].mac_tx_func); 3637 } 3638 3639 /* ARGSUSED */ 3640 static mac_tx_cookie_t 3641 mac_tx_single_ring_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3642 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3643 { 3644 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3645 mac_tx_stats_t stats; 3646 mac_tx_cookie_t cookie = 0; 3647 3648 ASSERT(srs_tx->st_mode == SRS_TX_DEFAULT); 3649 3650 /* Regular case with a single Tx ring */ 3651 /* 3652 * SRS_TX_BLOCKED is set when underlying NIC runs 3653 * out of Tx descs and messages start getting 3654 * queued. It won't get reset until 3655 * tx_srs_drain() completely drains out the 3656 * messages. 3657 */ 3658 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) { 3659 /* Tx descs/resources not available */ 3660 mutex_enter(&mac_srs->srs_lock); 3661 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) { 3662 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, 3663 flag, ret_mp); 3664 mutex_exit(&mac_srs->srs_lock); 3665 return (cookie); 3666 } 3667 /* 3668 * While we were computing mblk count, the 3669 * flow control condition got relieved. 3670 * Continue with the transmission. 3671 */ 3672 mutex_exit(&mac_srs->srs_lock); 3673 } 3674 3675 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 3676 mp_chain, &stats); 3677 3678 /* 3679 * Multiple threads could be here sending packets. 3680 * Under such conditions, it is not possible to 3681 * automically set SRS_TX_BLOCKED bit to indicate 3682 * out of tx desc condition. To atomically set 3683 * this, we queue the returned packet and do 3684 * the setting of SRS_TX_BLOCKED in 3685 * mac_tx_srs_drain(). 3686 */ 3687 if (mp_chain != NULL) { 3688 mutex_enter(&mac_srs->srs_lock); 3689 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, flag, ret_mp); 3690 mutex_exit(&mac_srs->srs_lock); 3691 return (cookie); 3692 } 3693 SRS_TX_STATS_UPDATE(mac_srs, &stats); 3694 3695 return (0); 3696 } 3697 3698 /* 3699 * mac_tx_serialize_mode 3700 * 3701 * This is an experimental mode implemented as per the request of PAE. 3702 * In this mode, all callers attempting to send a packet to the NIC 3703 * will get serialized. Only one thread at any time will access the 3704 * NIC to send the packet out. 3705 */ 3706 /* ARGSUSED */ 3707 static mac_tx_cookie_t 3708 mac_tx_serializer_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3709 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3710 { 3711 mac_tx_stats_t stats; 3712 mac_tx_cookie_t cookie = 0; 3713 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3714 3715 /* Single ring, serialize below */ 3716 ASSERT(srs_tx->st_mode == SRS_TX_SERIALIZE); 3717 mutex_enter(&mac_srs->srs_lock); 3718 if ((mac_srs->srs_first != NULL) || 3719 (mac_srs->srs_state & SRS_PROC)) { 3720 /* 3721 * In serialization mode, queue all packets until 3722 * TX_HIWAT is set. 3723 * If drop bit is set, drop if TX_HIWAT is set. 3724 * If no_enqueue is set, still enqueue until hiwat 3725 * is set and return mblks after TX_HIWAT is set. 3726 */ 3727 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, 3728 flag, 0, ret_mp); 3729 mutex_exit(&mac_srs->srs_lock); 3730 return (cookie); 3731 } 3732 /* 3733 * No packets queued, nothing on proc and no flow 3734 * control condition. Fast-path, ok. Do inline 3735 * processing. 3736 */ 3737 mac_srs->srs_state |= SRS_PROC; 3738 mutex_exit(&mac_srs->srs_lock); 3739 3740 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 3741 mp_chain, &stats); 3742 3743 mutex_enter(&mac_srs->srs_lock); 3744 mac_srs->srs_state &= ~SRS_PROC; 3745 if (mp_chain != NULL) { 3746 cookie = mac_tx_srs_enqueue(mac_srs, 3747 mp_chain, flag, 0, ret_mp); 3748 } 3749 if (mac_srs->srs_first != NULL) { 3750 /* 3751 * We processed inline our packet and a new 3752 * packet/s got queued while we were 3753 * processing. Wakeup srs worker 3754 */ 3755 cv_signal(&mac_srs->srs_async); 3756 } 3757 mutex_exit(&mac_srs->srs_lock); 3758 3759 if (cookie == 0) 3760 SRS_TX_STATS_UPDATE(mac_srs, &stats); 3761 3762 return (cookie); 3763 } 3764 3765 /* 3766 * mac_tx_fanout_mode 3767 * 3768 * In this mode, the SRS will have access to multiple Tx rings to send 3769 * the packet out. The fanout hint that is passed as an argument is 3770 * used to find an appropriate ring to fanout the traffic. Each Tx 3771 * ring, in turn, will have a soft ring associated with it. If a Tx 3772 * ring runs out of Tx desc's the returned packet will be queued in 3773 * the soft ring associated with that Tx ring. The srs itself will not 3774 * queue any packets. 3775 */ 3776 3777 #define MAC_TX_SOFT_RING_PROCESS(chain) { \ 3778 index = COMPUTE_INDEX(hash, mac_srs->srs_tx_ring_count), \ 3779 softring = mac_srs->srs_tx_soft_rings[index]; \ 3780 cookie = mac_tx_soft_ring_process(softring, chain, flag, ret_mp); \ 3781 DTRACE_PROBE2(tx__fanout, uint64_t, hash, uint_t, index); \ 3782 } 3783 3784 static mac_tx_cookie_t 3785 mac_tx_fanout_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3786 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3787 { 3788 mac_soft_ring_t *softring; 3789 uint64_t hash; 3790 uint_t index; 3791 mac_tx_cookie_t cookie = 0; 3792 3793 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT || 3794 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT); 3795 if (fanout_hint != 0) { 3796 /* 3797 * The hint is specified by the caller, simply pass the 3798 * whole chain to the soft ring. 3799 */ 3800 hash = HASH_HINT(fanout_hint); 3801 MAC_TX_SOFT_RING_PROCESS(mp_chain); 3802 } else { 3803 mblk_t *last_mp, *cur_mp, *sub_chain; 3804 uint64_t last_hash = 0; 3805 uint_t media = mac_srs->srs_mcip->mci_mip->mi_info.mi_media; 3806 3807 /* 3808 * Compute the hash from the contents (headers) of the 3809 * packets of the mblk chain. Split the chains into 3810 * subchains of the same conversation. 3811 * 3812 * Since there may be more than one ring used for 3813 * sub-chains of the same call, and since the caller 3814 * does not maintain per conversation state since it 3815 * passed a zero hint, unsent subchains will be 3816 * dropped. 3817 */ 3818 3819 flag |= MAC_DROP_ON_NO_DESC; 3820 ret_mp = NULL; 3821 3822 ASSERT(ret_mp == NULL); 3823 3824 sub_chain = NULL; 3825 last_mp = NULL; 3826 3827 for (cur_mp = mp_chain; cur_mp != NULL; 3828 cur_mp = cur_mp->b_next) { 3829 hash = mac_pkt_hash(media, cur_mp, MAC_PKT_HASH_L4, 3830 B_TRUE); 3831 if (last_hash != 0 && hash != last_hash) { 3832 /* 3833 * Starting a different subchain, send current 3834 * chain out. 3835 */ 3836 ASSERT(last_mp != NULL); 3837 last_mp->b_next = NULL; 3838 MAC_TX_SOFT_RING_PROCESS(sub_chain); 3839 sub_chain = NULL; 3840 } 3841 3842 /* add packet to subchain */ 3843 if (sub_chain == NULL) 3844 sub_chain = cur_mp; 3845 last_mp = cur_mp; 3846 last_hash = hash; 3847 } 3848 3849 if (sub_chain != NULL) { 3850 /* send last subchain */ 3851 ASSERT(last_mp != NULL); 3852 last_mp->b_next = NULL; 3853 MAC_TX_SOFT_RING_PROCESS(sub_chain); 3854 } 3855 3856 cookie = 0; 3857 } 3858 3859 return (cookie); 3860 } 3861 3862 /* 3863 * mac_tx_bw_mode 3864 * 3865 * In the bandwidth mode, Tx srs will allow packets to go down to Tx ring 3866 * only if bw is available. Otherwise the packets will be queued in 3867 * SRS. If the SRS has multiple Tx rings, then packets will get fanned 3868 * out to a Tx rings. 3869 */ 3870 static mac_tx_cookie_t 3871 mac_tx_bw_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3872 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3873 { 3874 int cnt, sz; 3875 mblk_t *tail; 3876 mac_tx_cookie_t cookie = 0; 3877 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3878 clock_t now; 3879 3880 ASSERT(TX_BANDWIDTH_MODE(mac_srs)); 3881 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL); 3882 mutex_enter(&mac_srs->srs_lock); 3883 if (mac_srs->srs_bw->mac_bw_limit == 0) { 3884 /* 3885 * zero bandwidth, no traffic is sent: drop the packets, 3886 * or return the whole chain if the caller requests all 3887 * unsent packets back. 3888 */ 3889 if (flag & MAC_TX_NO_ENQUEUE) { 3890 cookie = (mac_tx_cookie_t)mac_srs; 3891 *ret_mp = mp_chain; 3892 } else { 3893 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie); 3894 } 3895 mutex_exit(&mac_srs->srs_lock); 3896 return (cookie); 3897 } else if ((mac_srs->srs_first != NULL) || 3898 (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) { 3899 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag, 3900 fanout_hint, ret_mp); 3901 mutex_exit(&mac_srs->srs_lock); 3902 return (cookie); 3903 } 3904 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3905 now = ddi_get_lbolt(); 3906 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 3907 mac_srs->srs_bw->mac_bw_curr_time = now; 3908 mac_srs->srs_bw->mac_bw_used = 0; 3909 } else if (mac_srs->srs_bw->mac_bw_used > 3910 mac_srs->srs_bw->mac_bw_limit) { 3911 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 3912 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, 3913 mp_chain, tail, cnt, sz); 3914 /* 3915 * Wakeup worker thread. Note that worker 3916 * thread has to be woken up so that it 3917 * can fire up the timer to be woken up 3918 * on the next tick. Also once 3919 * BW_ENFORCED is set, it can only be 3920 * reset by srs_worker thread. Until then 3921 * all packets will get queued up in SRS 3922 * and hence this this code path won't be 3923 * entered until BW_ENFORCED is reset. 3924 */ 3925 cv_signal(&mac_srs->srs_async); 3926 mutex_exit(&mac_srs->srs_lock); 3927 return (cookie); 3928 } 3929 3930 mac_srs->srs_bw->mac_bw_used += sz; 3931 mutex_exit(&mac_srs->srs_lock); 3932 3933 if (srs_tx->st_mode == SRS_TX_BW_FANOUT) { 3934 mac_soft_ring_t *softring; 3935 uint_t indx, hash; 3936 3937 hash = HASH_HINT(fanout_hint); 3938 indx = COMPUTE_INDEX(hash, 3939 mac_srs->srs_tx_ring_count); 3940 softring = mac_srs->srs_tx_soft_rings[indx]; 3941 return (mac_tx_soft_ring_process(softring, mp_chain, flag, 3942 ret_mp)); 3943 } else if (srs_tx->st_mode == SRS_TX_BW_AGGR) { 3944 return (mac_tx_aggr_mode(mac_srs, mp_chain, 3945 fanout_hint, flag, ret_mp)); 3946 } else { 3947 mac_tx_stats_t stats; 3948 3949 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 3950 mp_chain, &stats); 3951 3952 if (mp_chain != NULL) { 3953 mutex_enter(&mac_srs->srs_lock); 3954 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 3955 if (mac_srs->srs_bw->mac_bw_used > sz) 3956 mac_srs->srs_bw->mac_bw_used -= sz; 3957 else 3958 mac_srs->srs_bw->mac_bw_used = 0; 3959 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag, 3960 fanout_hint, ret_mp); 3961 mutex_exit(&mac_srs->srs_lock); 3962 return (cookie); 3963 } 3964 SRS_TX_STATS_UPDATE(mac_srs, &stats); 3965 3966 return (0); 3967 } 3968 } 3969 3970 /* 3971 * mac_tx_aggr_mode 3972 * 3973 * This routine invokes an aggr function, aggr_find_tx_ring(), to find 3974 * a (pseudo) Tx ring belonging to a port on which the packet has to 3975 * be sent. aggr_find_tx_ring() first finds the outgoing port based on 3976 * L2/L3/L4 policy and then uses the fanout_hint passed to it to pick 3977 * a Tx ring from the selected port. 3978 * 3979 * Note that a port can be deleted from the aggregation. In such a case, 3980 * the aggregation layer first separates the port from the rest of the 3981 * ports making sure that port (and thus any Tx rings associated with 3982 * it) won't get selected in the call to aggr_find_tx_ring() function. 3983 * Later calls are made to mac_group_rem_ring() passing pseudo Tx ring 3984 * handles one by one which in turn will quiesce the Tx SRS and remove 3985 * the soft ring associated with the pseudo Tx ring. Unlike Rx side 3986 * where a cookie is used to protect against mac_rx_ring() calls on 3987 * rings that have been removed, no such cookie is needed on the Tx 3988 * side as the pseudo Tx ring won't be available anymore to 3989 * aggr_find_tx_ring() once the port has been removed. 3990 */ 3991 static mac_tx_cookie_t 3992 mac_tx_aggr_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain, 3993 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp) 3994 { 3995 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 3996 mac_tx_ring_fn_t find_tx_ring_fn; 3997 mac_ring_handle_t ring = NULL; 3998 void *arg; 3999 mac_soft_ring_t *sringp; 4000 4001 find_tx_ring_fn = srs_tx->st_capab_aggr.mca_find_tx_ring_fn; 4002 arg = srs_tx->st_capab_aggr.mca_arg; 4003 if (find_tx_ring_fn(arg, mp_chain, fanout_hint, &ring) == NULL) 4004 return (0); 4005 sringp = srs_tx->st_soft_rings[((mac_ring_t *)ring)->mr_index]; 4006 return (mac_tx_soft_ring_process(sringp, mp_chain, flag, ret_mp)); 4007 } 4008 4009 void 4010 mac_tx_invoke_callbacks(mac_client_impl_t *mcip, mac_tx_cookie_t cookie) 4011 { 4012 mac_cb_t *mcb; 4013 mac_tx_notify_cb_t *mtnfp; 4014 4015 /* Wakeup callback registered clients */ 4016 MAC_CALLBACK_WALKER_INC(&mcip->mci_tx_notify_cb_info); 4017 for (mcb = mcip->mci_tx_notify_cb_list; mcb != NULL; 4018 mcb = mcb->mcb_nextp) { 4019 mtnfp = (mac_tx_notify_cb_t *)mcb->mcb_objp; 4020 mtnfp->mtnf_fn(mtnfp->mtnf_arg, cookie); 4021 } 4022 MAC_CALLBACK_WALKER_DCR(&mcip->mci_tx_notify_cb_info, 4023 &mcip->mci_tx_notify_cb_list); 4024 } 4025 4026 /* ARGSUSED */ 4027 void 4028 mac_tx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type) 4029 { 4030 mblk_t *head, *tail; 4031 size_t sz; 4032 uint32_t tx_mode; 4033 uint_t saved_pkt_count; 4034 mac_tx_stats_t stats; 4035 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 4036 clock_t now; 4037 4038 saved_pkt_count = 0; 4039 ASSERT(mutex_owned(&mac_srs->srs_lock)); 4040 ASSERT(!(mac_srs->srs_state & SRS_PROC)); 4041 4042 mac_srs->srs_state |= SRS_PROC; 4043 4044 tx_mode = srs_tx->st_mode; 4045 if (tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_SERIALIZE) { 4046 if (mac_srs->srs_first != NULL) { 4047 head = mac_srs->srs_first; 4048 tail = mac_srs->srs_last; 4049 saved_pkt_count = mac_srs->srs_count; 4050 mac_srs->srs_first = NULL; 4051 mac_srs->srs_last = NULL; 4052 mac_srs->srs_count = 0; 4053 mutex_exit(&mac_srs->srs_lock); 4054 4055 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 4056 head, &stats); 4057 4058 mutex_enter(&mac_srs->srs_lock); 4059 if (head != NULL) { 4060 /* Device out of tx desc, set block */ 4061 if (head->b_next == NULL) 4062 VERIFY(head == tail); 4063 tail->b_next = mac_srs->srs_first; 4064 mac_srs->srs_first = head; 4065 mac_srs->srs_count += 4066 (saved_pkt_count - stats.mts_opackets); 4067 if (mac_srs->srs_last == NULL) 4068 mac_srs->srs_last = tail; 4069 MAC_TX_SRS_BLOCK(mac_srs, head); 4070 } else { 4071 srs_tx->st_woken_up = B_FALSE; 4072 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4073 } 4074 } 4075 } else if (tx_mode == SRS_TX_BW) { 4076 /* 4077 * We are here because the timer fired and we have some data 4078 * to tranmit. Also mac_tx_srs_worker should have reset 4079 * SRS_BW_ENFORCED flag 4080 */ 4081 ASSERT(!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)); 4082 head = tail = mac_srs->srs_first; 4083 while (mac_srs->srs_first != NULL) { 4084 tail = mac_srs->srs_first; 4085 tail->b_prev = NULL; 4086 mac_srs->srs_first = tail->b_next; 4087 if (mac_srs->srs_first == NULL) 4088 mac_srs->srs_last = NULL; 4089 mac_srs->srs_count--; 4090 sz = msgdsize(tail); 4091 mac_srs->srs_size -= sz; 4092 saved_pkt_count++; 4093 MAC_TX_UPDATE_BW_INFO(mac_srs, sz); 4094 4095 if (mac_srs->srs_bw->mac_bw_used < 4096 mac_srs->srs_bw->mac_bw_limit) 4097 continue; 4098 4099 now = ddi_get_lbolt(); 4100 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 4101 mac_srs->srs_bw->mac_bw_curr_time = now; 4102 mac_srs->srs_bw->mac_bw_used = sz; 4103 continue; 4104 } 4105 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 4106 break; 4107 } 4108 4109 ASSERT((head == NULL && tail == NULL) || 4110 (head != NULL && tail != NULL)); 4111 if (tail != NULL) { 4112 tail->b_next = NULL; 4113 mutex_exit(&mac_srs->srs_lock); 4114 4115 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2, 4116 head, &stats); 4117 4118 mutex_enter(&mac_srs->srs_lock); 4119 if (head != NULL) { 4120 uint_t size_sent; 4121 4122 /* Device out of tx desc, set block */ 4123 if (head->b_next == NULL) 4124 VERIFY(head == tail); 4125 tail->b_next = mac_srs->srs_first; 4126 mac_srs->srs_first = head; 4127 mac_srs->srs_count += 4128 (saved_pkt_count - stats.mts_opackets); 4129 if (mac_srs->srs_last == NULL) 4130 mac_srs->srs_last = tail; 4131 size_sent = sz - stats.mts_obytes; 4132 mac_srs->srs_size += size_sent; 4133 mac_srs->srs_bw->mac_bw_sz += size_sent; 4134 if (mac_srs->srs_bw->mac_bw_used > size_sent) { 4135 mac_srs->srs_bw->mac_bw_used -= 4136 size_sent; 4137 } else { 4138 mac_srs->srs_bw->mac_bw_used = 0; 4139 } 4140 MAC_TX_SRS_BLOCK(mac_srs, head); 4141 } else { 4142 srs_tx->st_woken_up = B_FALSE; 4143 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4144 } 4145 } 4146 } else if (tx_mode == SRS_TX_BW_FANOUT || tx_mode == SRS_TX_BW_AGGR) { 4147 mblk_t *prev; 4148 uint64_t hint; 4149 4150 /* 4151 * We are here because the timer fired and we 4152 * have some quota to tranmit. 4153 */ 4154 prev = NULL; 4155 head = tail = mac_srs->srs_first; 4156 while (mac_srs->srs_first != NULL) { 4157 tail = mac_srs->srs_first; 4158 mac_srs->srs_first = tail->b_next; 4159 if (mac_srs->srs_first == NULL) 4160 mac_srs->srs_last = NULL; 4161 mac_srs->srs_count--; 4162 sz = msgdsize(tail); 4163 mac_srs->srs_size -= sz; 4164 mac_srs->srs_bw->mac_bw_used += sz; 4165 if (prev == NULL) 4166 hint = (ulong_t)tail->b_prev; 4167 if (hint != (ulong_t)tail->b_prev) { 4168 prev->b_next = NULL; 4169 mutex_exit(&mac_srs->srs_lock); 4170 TX_SRS_TO_SOFT_RING(mac_srs, head, hint); 4171 head = tail; 4172 hint = (ulong_t)tail->b_prev; 4173 mutex_enter(&mac_srs->srs_lock); 4174 } 4175 4176 prev = tail; 4177 tail->b_prev = NULL; 4178 if (mac_srs->srs_bw->mac_bw_used < 4179 mac_srs->srs_bw->mac_bw_limit) 4180 continue; 4181 4182 now = ddi_get_lbolt(); 4183 if (mac_srs->srs_bw->mac_bw_curr_time != now) { 4184 mac_srs->srs_bw->mac_bw_curr_time = now; 4185 mac_srs->srs_bw->mac_bw_used = 0; 4186 continue; 4187 } 4188 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED; 4189 break; 4190 } 4191 ASSERT((head == NULL && tail == NULL) || 4192 (head != NULL && tail != NULL)); 4193 if (tail != NULL) { 4194 tail->b_next = NULL; 4195 mutex_exit(&mac_srs->srs_lock); 4196 TX_SRS_TO_SOFT_RING(mac_srs, head, hint); 4197 mutex_enter(&mac_srs->srs_lock); 4198 } 4199 } 4200 /* 4201 * SRS_TX_FANOUT case not considered here because packets 4202 * won't be queued in the SRS for this case. Packets will 4203 * be sent directly to soft rings underneath and if there 4204 * is any queueing at all, it would be in Tx side soft 4205 * rings. 4206 */ 4207 4208 /* 4209 * When srs_count becomes 0, reset SRS_TX_HIWAT and 4210 * SRS_TX_WAKEUP_CLIENT and wakeup registered clients. 4211 */ 4212 if (mac_srs->srs_count == 0 && (mac_srs->srs_state & 4213 (SRS_TX_HIWAT | SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED))) { 4214 mac_client_impl_t *mcip = mac_srs->srs_mcip; 4215 boolean_t wakeup_required = B_FALSE; 4216 4217 if (mac_srs->srs_state & 4218 (SRS_TX_HIWAT|SRS_TX_WAKEUP_CLIENT)) { 4219 wakeup_required = B_TRUE; 4220 } 4221 mac_srs->srs_state &= ~(SRS_TX_HIWAT | 4222 SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED); 4223 mutex_exit(&mac_srs->srs_lock); 4224 if (wakeup_required) { 4225 mac_tx_invoke_callbacks(mcip, (mac_tx_cookie_t)mac_srs); 4226 /* 4227 * If the client is not the primary MAC client, then we 4228 * need to send the notification to the clients upper 4229 * MAC, i.e. mci_upper_mip. 4230 */ 4231 mac_tx_notify(mcip->mci_upper_mip != NULL ? 4232 mcip->mci_upper_mip : mcip->mci_mip); 4233 } 4234 mutex_enter(&mac_srs->srs_lock); 4235 } 4236 mac_srs->srs_state &= ~SRS_PROC; 4237 } 4238 4239 /* 4240 * Given a packet, get the flow_entry that identifies the flow 4241 * to which that packet belongs. The flow_entry will contain 4242 * the transmit function to be used to send the packet. If the 4243 * function returns NULL, the packet should be sent using the 4244 * underlying NIC. 4245 */ 4246 static flow_entry_t * 4247 mac_tx_classify(mac_impl_t *mip, mblk_t *mp) 4248 { 4249 flow_entry_t *flent = NULL; 4250 mac_client_impl_t *mcip; 4251 int err; 4252 4253 /* 4254 * Do classification on the packet. 4255 */ 4256 err = mac_flow_lookup(mip->mi_flow_tab, mp, FLOW_OUTBOUND, &flent); 4257 if (err != 0) 4258 return (NULL); 4259 4260 /* 4261 * This flent might just be an additional one on the MAC client, 4262 * i.e. for classification purposes (different fdesc), however 4263 * the resources, SRS et. al., are in the mci_flent, so if 4264 * this isn't the mci_flent, we need to get it. 4265 */ 4266 if ((mcip = flent->fe_mcip) != NULL && mcip->mci_flent != flent) { 4267 FLOW_REFRELE(flent); 4268 flent = mcip->mci_flent; 4269 FLOW_TRY_REFHOLD(flent, err); 4270 if (err != 0) 4271 return (NULL); 4272 } 4273 4274 return (flent); 4275 } 4276 4277 /* 4278 * This macro is only meant to be used by mac_tx_send(). 4279 */ 4280 #define CHECK_VID_AND_ADD_TAG(mp) { \ 4281 if (vid_check) { \ 4282 int err = 0; \ 4283 \ 4284 MAC_VID_CHECK(src_mcip, (mp), err); \ 4285 if (err != 0) { \ 4286 freemsg((mp)); \ 4287 (mp) = next; \ 4288 oerrors++; \ 4289 continue; \ 4290 } \ 4291 } \ 4292 if (add_tag) { \ 4293 (mp) = mac_add_vlan_tag((mp), 0, vid); \ 4294 if ((mp) == NULL) { \ 4295 (mp) = next; \ 4296 oerrors++; \ 4297 continue; \ 4298 } \ 4299 } \ 4300 } 4301 4302 mblk_t * 4303 mac_tx_send(mac_client_handle_t mch, mac_ring_handle_t ring, mblk_t *mp_chain, 4304 mac_tx_stats_t *stats) 4305 { 4306 mac_client_impl_t *src_mcip = (mac_client_impl_t *)mch; 4307 mac_impl_t *mip = src_mcip->mci_mip; 4308 uint_t obytes = 0, opackets = 0, oerrors = 0; 4309 mblk_t *mp = NULL, *next; 4310 boolean_t vid_check, add_tag; 4311 uint16_t vid = 0; 4312 4313 if (mip->mi_nclients > 1) { 4314 vid_check = MAC_VID_CHECK_NEEDED(src_mcip); 4315 add_tag = MAC_TAG_NEEDED(src_mcip); 4316 if (add_tag) 4317 vid = mac_client_vid(mch); 4318 } else { 4319 ASSERT(mip->mi_nclients == 1); 4320 vid_check = add_tag = B_FALSE; 4321 } 4322 4323 /* 4324 * Fastpath: if there's only one client, we simply send 4325 * the packet down to the underlying NIC. 4326 */ 4327 if (mip->mi_nactiveclients == 1) { 4328 DTRACE_PROBE2(fastpath, 4329 mac_client_impl_t *, src_mcip, mblk_t *, mp_chain); 4330 4331 mp = mp_chain; 4332 while (mp != NULL) { 4333 next = mp->b_next; 4334 mp->b_next = NULL; 4335 opackets++; 4336 obytes += (mp->b_cont == NULL ? MBLKL(mp) : 4337 msgdsize(mp)); 4338 4339 CHECK_VID_AND_ADD_TAG(mp); 4340 MAC_TX(mip, ring, mp, src_mcip); 4341 4342 /* 4343 * If the driver is out of descriptors and does a 4344 * partial send it will return a chain of unsent 4345 * mblks. Adjust the accounting stats. 4346 */ 4347 if (mp != NULL) { 4348 opackets--; 4349 obytes -= msgdsize(mp); 4350 mp->b_next = next; 4351 break; 4352 } 4353 mp = next; 4354 } 4355 goto done; 4356 } 4357 4358 /* 4359 * No fastpath, we either have more than one MAC client 4360 * defined on top of the same MAC, or one or more MAC 4361 * client promiscuous callbacks. 4362 */ 4363 DTRACE_PROBE3(slowpath, mac_client_impl_t *, 4364 src_mcip, int, mip->mi_nclients, mblk_t *, mp_chain); 4365 4366 mp = mp_chain; 4367 while (mp != NULL) { 4368 flow_entry_t *dst_flow_ent; 4369 void *flow_cookie; 4370 size_t pkt_size; 4371 mblk_t *mp1; 4372 4373 next = mp->b_next; 4374 mp->b_next = NULL; 4375 opackets++; 4376 pkt_size = (mp->b_cont == NULL ? MBLKL(mp) : msgdsize(mp)); 4377 obytes += pkt_size; 4378 CHECK_VID_AND_ADD_TAG(mp); 4379 4380 /* 4381 * Find the destination. 4382 */ 4383 dst_flow_ent = mac_tx_classify(mip, mp); 4384 4385 if (dst_flow_ent != NULL) { 4386 size_t hdrsize; 4387 int err = 0; 4388 4389 if (mip->mi_info.mi_nativemedia == DL_ETHER) { 4390 struct ether_vlan_header *evhp = 4391 (struct ether_vlan_header *)mp->b_rptr; 4392 4393 if (ntohs(evhp->ether_tpid) == ETHERTYPE_VLAN) 4394 hdrsize = sizeof (*evhp); 4395 else 4396 hdrsize = sizeof (struct ether_header); 4397 } else { 4398 mac_header_info_t mhi; 4399 4400 err = mac_header_info((mac_handle_t)mip, 4401 mp, &mhi); 4402 if (err == 0) 4403 hdrsize = mhi.mhi_hdrsize; 4404 } 4405 4406 /* 4407 * Got a matching flow. It's either another 4408 * MAC client, or a broadcast/multicast flow. 4409 * Make sure the packet size is within the 4410 * allowed size. If not drop the packet and 4411 * move to next packet. 4412 */ 4413 if (err != 0 || 4414 (pkt_size - hdrsize) > mip->mi_sdu_max) { 4415 oerrors++; 4416 DTRACE_PROBE2(loopback__drop, size_t, pkt_size, 4417 mblk_t *, mp); 4418 freemsg(mp); 4419 mp = next; 4420 FLOW_REFRELE(dst_flow_ent); 4421 continue; 4422 } 4423 flow_cookie = mac_flow_get_client_cookie(dst_flow_ent); 4424 if (flow_cookie != NULL) { 4425 /* 4426 * The vnic_bcast_send function expects 4427 * to receive the sender MAC client 4428 * as value for arg2. 4429 */ 4430 mac_bcast_send(flow_cookie, src_mcip, mp, 4431 B_TRUE); 4432 } else { 4433 /* 4434 * loopback the packet to a local MAC 4435 * client. We force a context switch 4436 * if both source and destination MAC 4437 * clients are used by IP, i.e. 4438 * bypass is set. 4439 */ 4440 boolean_t do_switch; 4441 mac_client_impl_t *dst_mcip = 4442 dst_flow_ent->fe_mcip; 4443 4444 /* 4445 * Check if there are promiscuous mode 4446 * callbacks defined. This check is 4447 * done here in the 'else' case and 4448 * not in other cases because this 4449 * path is for local loopback 4450 * communication which does not go 4451 * through MAC_TX(). For paths that go 4452 * through MAC_TX(), the promisc_list 4453 * check is done inside the MAC_TX() 4454 * macro. 4455 */ 4456 if (mip->mi_promisc_list != NULL) 4457 mac_promisc_dispatch(mip, mp, src_mcip); 4458 4459 do_switch = ((src_mcip->mci_state_flags & 4460 dst_mcip->mci_state_flags & 4461 MCIS_CLIENT_POLL_CAPABLE) != 0); 4462 4463 if ((mp1 = mac_fix_cksum(mp)) != NULL) { 4464 (dst_flow_ent->fe_cb_fn)( 4465 dst_flow_ent->fe_cb_arg1, 4466 dst_flow_ent->fe_cb_arg2, 4467 mp1, do_switch); 4468 } 4469 } 4470 FLOW_REFRELE(dst_flow_ent); 4471 } else { 4472 /* 4473 * Unknown destination, send via the underlying 4474 * NIC. 4475 */ 4476 MAC_TX(mip, ring, mp, src_mcip); 4477 if (mp != NULL) { 4478 /* 4479 * Adjust for the last packet that 4480 * could not be transmitted 4481 */ 4482 opackets--; 4483 obytes -= pkt_size; 4484 mp->b_next = next; 4485 break; 4486 } 4487 } 4488 mp = next; 4489 } 4490 4491 done: 4492 stats->mts_obytes = obytes; 4493 stats->mts_opackets = opackets; 4494 stats->mts_oerrors = oerrors; 4495 return (mp); 4496 } 4497 4498 /* 4499 * mac_tx_srs_ring_present 4500 * 4501 * Returns whether the specified ring is part of the specified SRS. 4502 */ 4503 boolean_t 4504 mac_tx_srs_ring_present(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring) 4505 { 4506 int i; 4507 mac_soft_ring_t *soft_ring; 4508 4509 if (srs->srs_tx.st_arg2 == tx_ring) 4510 return (B_TRUE); 4511 4512 for (i = 0; i < srs->srs_tx_ring_count; i++) { 4513 soft_ring = srs->srs_tx_soft_rings[i]; 4514 if (soft_ring->s_ring_tx_arg2 == tx_ring) 4515 return (B_TRUE); 4516 } 4517 4518 return (B_FALSE); 4519 } 4520 4521 /* 4522 * mac_tx_srs_get_soft_ring 4523 * 4524 * Returns the TX soft ring associated with the given ring, if present. 4525 */ 4526 mac_soft_ring_t * 4527 mac_tx_srs_get_soft_ring(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring) 4528 { 4529 int i; 4530 mac_soft_ring_t *soft_ring; 4531 4532 if (srs->srs_tx.st_arg2 == tx_ring) 4533 return (NULL); 4534 4535 for (i = 0; i < srs->srs_tx_ring_count; i++) { 4536 soft_ring = srs->srs_tx_soft_rings[i]; 4537 if (soft_ring->s_ring_tx_arg2 == tx_ring) 4538 return (soft_ring); 4539 } 4540 4541 return (NULL); 4542 } 4543 4544 /* 4545 * mac_tx_srs_wakeup 4546 * 4547 * Called when Tx desc become available. Wakeup the appropriate worker 4548 * thread after resetting the SRS_TX_BLOCKED/S_RING_BLOCK bit in the 4549 * state field. 4550 */ 4551 void 4552 mac_tx_srs_wakeup(mac_soft_ring_set_t *mac_srs, mac_ring_handle_t ring) 4553 { 4554 int i; 4555 mac_soft_ring_t *sringp; 4556 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx; 4557 4558 mutex_enter(&mac_srs->srs_lock); 4559 /* 4560 * srs_tx_ring_count == 0 is the single ring mode case. In 4561 * this mode, there will not be Tx soft rings associated 4562 * with the SRS. 4563 */ 4564 if (!MAC_TX_SOFT_RINGS(mac_srs)) { 4565 if (srs_tx->st_arg2 == ring && 4566 mac_srs->srs_state & SRS_TX_BLOCKED) { 4567 mac_srs->srs_state &= ~SRS_TX_BLOCKED; 4568 srs_tx->st_stat.mts_unblockcnt++; 4569 cv_signal(&mac_srs->srs_async); 4570 } 4571 /* 4572 * A wakeup can come before tx_srs_drain() could 4573 * grab srs lock and set SRS_TX_BLOCKED. So 4574 * always set woken_up flag when we come here. 4575 */ 4576 srs_tx->st_woken_up = B_TRUE; 4577 mutex_exit(&mac_srs->srs_lock); 4578 return; 4579 } 4580 4581 /* 4582 * If you are here, it is for FANOUT, BW_FANOUT, 4583 * AGGR_MODE or AGGR_BW_MODE case 4584 */ 4585 for (i = 0; i < mac_srs->srs_tx_ring_count; i++) { 4586 sringp = mac_srs->srs_tx_soft_rings[i]; 4587 mutex_enter(&sringp->s_ring_lock); 4588 if (sringp->s_ring_tx_arg2 == ring) { 4589 if (sringp->s_ring_state & S_RING_BLOCK) { 4590 sringp->s_ring_state &= ~S_RING_BLOCK; 4591 sringp->s_st_stat.mts_unblockcnt++; 4592 cv_signal(&sringp->s_ring_async); 4593 } 4594 sringp->s_ring_tx_woken_up = B_TRUE; 4595 } 4596 mutex_exit(&sringp->s_ring_lock); 4597 } 4598 mutex_exit(&mac_srs->srs_lock); 4599 } 4600 4601 /* 4602 * Once the driver is done draining, send a MAC_NOTE_TX notification to unleash 4603 * the blocked clients again. 4604 */ 4605 void 4606 mac_tx_notify(mac_impl_t *mip) 4607 { 4608 i_mac_notify(mip, MAC_NOTE_TX); 4609 } 4610 4611 /* 4612 * RX SOFTRING RELATED FUNCTIONS 4613 * 4614 * These functions really belong in mac_soft_ring.c and here for 4615 * a short period. 4616 */ 4617 4618 #define SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \ 4619 /* \ 4620 * Enqueue our mblk chain. \ 4621 */ \ 4622 ASSERT(MUTEX_HELD(&(ringp)->s_ring_lock)); \ 4623 \ 4624 if ((ringp)->s_ring_last != NULL) \ 4625 (ringp)->s_ring_last->b_next = (mp); \ 4626 else \ 4627 (ringp)->s_ring_first = (mp); \ 4628 (ringp)->s_ring_last = (tail); \ 4629 (ringp)->s_ring_count += (cnt); \ 4630 ASSERT((ringp)->s_ring_count > 0); \ 4631 if ((ringp)->s_ring_type & ST_RING_BW_CTL) { \ 4632 (ringp)->s_ring_size += sz; \ 4633 } \ 4634 } 4635 4636 /* 4637 * Default entry point to deliver a packet chain to a MAC client. 4638 * If the MAC client has flows, do the classification with these 4639 * flows as well. 4640 */ 4641 /* ARGSUSED */ 4642 void 4643 mac_rx_deliver(void *arg1, mac_resource_handle_t mrh, mblk_t *mp_chain, 4644 mac_header_info_t *arg3) 4645 { 4646 mac_client_impl_t *mcip = arg1; 4647 4648 if (mcip->mci_nvids == 1 && 4649 !(mcip->mci_state_flags & MCIS_STRIP_DISABLE)) { 4650 /* 4651 * If the client has exactly one VID associated with it 4652 * and striping of VLAN header is not disabled, 4653 * remove the VLAN tag from the packet before 4654 * passing it on to the client's receive callback. 4655 * Note that this needs to be done after we dispatch 4656 * the packet to the promiscuous listeners of the 4657 * client, since they expect to see the whole 4658 * frame including the VLAN headers. 4659 */ 4660 mp_chain = mac_strip_vlan_tag_chain(mp_chain); 4661 } 4662 4663 mcip->mci_rx_fn(mcip->mci_rx_arg, mrh, mp_chain, B_FALSE); 4664 } 4665 4666 /* 4667 * mac_rx_soft_ring_process 4668 * 4669 * process a chain for a given soft ring. The number of packets queued 4670 * in the SRS and its associated soft rings (including this one) is 4671 * very small (tracked by srs_poll_pkt_cnt), then allow the entering 4672 * thread (interrupt or poll thread) to do inline processing. This 4673 * helps keep the latency down under low load. 4674 * 4675 * The proc and arg for each mblk is already stored in the mblk in 4676 * appropriate places. 4677 */ 4678 /* ARGSUSED */ 4679 void 4680 mac_rx_soft_ring_process(mac_client_impl_t *mcip, mac_soft_ring_t *ringp, 4681 mblk_t *mp_chain, mblk_t *tail, int cnt, size_t sz) 4682 { 4683 mac_direct_rx_t proc; 4684 void *arg1; 4685 mac_resource_handle_t arg2; 4686 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4687 4688 ASSERT(ringp != NULL); 4689 ASSERT(mp_chain != NULL); 4690 ASSERT(tail != NULL); 4691 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4692 4693 mutex_enter(&ringp->s_ring_lock); 4694 ringp->s_ring_total_inpkt += cnt; 4695 ringp->s_ring_total_rbytes += sz; 4696 if ((mac_srs->srs_rx.sr_poll_pkt_cnt <= 1) && 4697 !(ringp->s_ring_type & ST_RING_WORKER_ONLY)) { 4698 /* If on processor or blanking on, then enqueue and return */ 4699 if (ringp->s_ring_state & S_RING_BLANK || 4700 ringp->s_ring_state & S_RING_PROC) { 4701 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4702 mutex_exit(&ringp->s_ring_lock); 4703 return; 4704 } 4705 proc = ringp->s_ring_rx_func; 4706 arg1 = ringp->s_ring_rx_arg1; 4707 arg2 = ringp->s_ring_rx_arg2; 4708 /* 4709 * See if anything is already queued. If we are the 4710 * first packet, do inline processing else queue the 4711 * packet and do the drain. 4712 */ 4713 if (ringp->s_ring_first == NULL) { 4714 /* 4715 * Fast-path, ok to process and nothing queued. 4716 */ 4717 ringp->s_ring_run = curthread; 4718 ringp->s_ring_state |= (S_RING_PROC); 4719 4720 mutex_exit(&ringp->s_ring_lock); 4721 4722 /* 4723 * We are the chain of 1 packet so 4724 * go through this fast path. 4725 */ 4726 ASSERT(mp_chain->b_next == NULL); 4727 4728 (*proc)(arg1, arg2, mp_chain, NULL); 4729 4730 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4731 /* 4732 * If we have a soft ring set which is doing 4733 * bandwidth control, we need to decrement 4734 * srs_size and count so it the SRS can have a 4735 * accurate idea of what is the real data 4736 * queued between SRS and its soft rings. We 4737 * decrement the counters only when the packet 4738 * gets processed by both SRS and the soft ring. 4739 */ 4740 mutex_enter(&mac_srs->srs_lock); 4741 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt); 4742 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz); 4743 mutex_exit(&mac_srs->srs_lock); 4744 4745 mutex_enter(&ringp->s_ring_lock); 4746 ringp->s_ring_run = NULL; 4747 ringp->s_ring_state &= ~S_RING_PROC; 4748 if (ringp->s_ring_state & S_RING_CLIENT_WAIT) 4749 cv_signal(&ringp->s_ring_client_cv); 4750 4751 if ((ringp->s_ring_first == NULL) || 4752 (ringp->s_ring_state & S_RING_BLANK)) { 4753 /* 4754 * We processed inline our packet and 4755 * nothing new has arrived or our 4756 * receiver doesn't want to receive 4757 * any packets. We are done. 4758 */ 4759 mutex_exit(&ringp->s_ring_lock); 4760 return; 4761 } 4762 } else { 4763 SOFT_RING_ENQUEUE_CHAIN(ringp, 4764 mp_chain, tail, cnt, sz); 4765 } 4766 4767 /* 4768 * We are here because either we couldn't do inline 4769 * processing (because something was already 4770 * queued), or we had a chain of more than one 4771 * packet, or something else arrived after we were 4772 * done with inline processing. 4773 */ 4774 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); 4775 ASSERT(ringp->s_ring_first != NULL); 4776 4777 ringp->s_ring_drain_func(ringp); 4778 mutex_exit(&ringp->s_ring_lock); 4779 return; 4780 } else { 4781 /* ST_RING_WORKER_ONLY case */ 4782 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4783 mac_soft_ring_worker_wakeup(ringp); 4784 mutex_exit(&ringp->s_ring_lock); 4785 } 4786 } 4787 4788 /* 4789 * TX SOFTRING RELATED FUNCTIONS 4790 * 4791 * These functions really belong in mac_soft_ring.c and here for 4792 * a short period. 4793 */ 4794 4795 #define TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \ 4796 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); \ 4797 ringp->s_ring_state |= S_RING_ENQUEUED; \ 4798 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); \ 4799 } 4800 4801 /* 4802 * mac_tx_sring_queued 4803 * 4804 * When we are out of transmit descriptors and we already have a 4805 * queue that exceeds hiwat (or the client called us with 4806 * MAC_TX_NO_ENQUEUE or MAC_DROP_ON_NO_DESC flag), return the 4807 * soft ring pointer as the opaque cookie for the client enable 4808 * flow control. 4809 */ 4810 static mac_tx_cookie_t 4811 mac_tx_sring_enqueue(mac_soft_ring_t *ringp, mblk_t *mp_chain, uint16_t flag, 4812 mblk_t **ret_mp) 4813 { 4814 int cnt; 4815 size_t sz; 4816 mblk_t *tail; 4817 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4818 mac_tx_cookie_t cookie = 0; 4819 boolean_t wakeup_worker = B_TRUE; 4820 4821 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); 4822 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 4823 if (flag & MAC_DROP_ON_NO_DESC) { 4824 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE); 4825 /* increment freed stats */ 4826 ringp->s_ring_drops += cnt; 4827 cookie = (mac_tx_cookie_t)ringp; 4828 } else { 4829 if (ringp->s_ring_first != NULL) 4830 wakeup_worker = B_FALSE; 4831 4832 if (flag & MAC_TX_NO_ENQUEUE) { 4833 /* 4834 * If QUEUED is not set, queue the packet 4835 * and let mac_tx_soft_ring_drain() set 4836 * the TX_BLOCKED bit for the reasons 4837 * explained above. Otherwise, return the 4838 * mblks. 4839 */ 4840 if (wakeup_worker) { 4841 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, 4842 mp_chain, tail, cnt, sz); 4843 } else { 4844 ringp->s_ring_state |= S_RING_WAKEUP_CLIENT; 4845 cookie = (mac_tx_cookie_t)ringp; 4846 *ret_mp = mp_chain; 4847 } 4848 } else { 4849 boolean_t enqueue = B_TRUE; 4850 4851 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) { 4852 /* 4853 * flow-controlled. Store ringp in cookie 4854 * so that it can be returned as 4855 * mac_tx_cookie_t to client 4856 */ 4857 ringp->s_ring_state |= S_RING_TX_HIWAT; 4858 cookie = (mac_tx_cookie_t)ringp; 4859 ringp->s_ring_hiwat_cnt++; 4860 if (ringp->s_ring_count > 4861 ringp->s_ring_tx_max_q_cnt) { 4862 /* increment freed stats */ 4863 ringp->s_ring_drops += cnt; 4864 /* 4865 * b_prev may be set to the fanout hint 4866 * hence can't use freemsg directly 4867 */ 4868 mac_pkt_drop(NULL, NULL, 4869 mp_chain, B_FALSE); 4870 DTRACE_PROBE1(tx_queued_hiwat, 4871 mac_soft_ring_t *, ringp); 4872 enqueue = B_FALSE; 4873 } 4874 } 4875 if (enqueue) { 4876 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, 4877 tail, cnt, sz); 4878 } 4879 } 4880 if (wakeup_worker) 4881 cv_signal(&ringp->s_ring_async); 4882 } 4883 return (cookie); 4884 } 4885 4886 4887 /* 4888 * mac_tx_soft_ring_process 4889 * 4890 * This routine is called when fanning out outgoing traffic among 4891 * multipe Tx rings. 4892 * Note that a soft ring is associated with a h/w Tx ring. 4893 */ 4894 mac_tx_cookie_t 4895 mac_tx_soft_ring_process(mac_soft_ring_t *ringp, mblk_t *mp_chain, 4896 uint16_t flag, mblk_t **ret_mp) 4897 { 4898 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set; 4899 int cnt; 4900 size_t sz; 4901 mblk_t *tail; 4902 mac_tx_cookie_t cookie = 0; 4903 4904 ASSERT(ringp != NULL); 4905 ASSERT(mp_chain != NULL); 4906 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock)); 4907 /* 4908 * The following modes can come here: SRS_TX_BW_FANOUT, 4909 * SRS_TX_FANOUT, SRS_TX_AGGR, SRS_TX_BW_AGGR. 4910 */ 4911 ASSERT(MAC_TX_SOFT_RINGS(mac_srs)); 4912 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT || 4913 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT || 4914 mac_srs->srs_tx.st_mode == SRS_TX_AGGR || 4915 mac_srs->srs_tx.st_mode == SRS_TX_BW_AGGR); 4916 4917 if (ringp->s_ring_type & ST_RING_WORKER_ONLY) { 4918 /* Serialization mode */ 4919 4920 mutex_enter(&ringp->s_ring_lock); 4921 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) { 4922 cookie = mac_tx_sring_enqueue(ringp, mp_chain, 4923 flag, ret_mp); 4924 mutex_exit(&ringp->s_ring_lock); 4925 return (cookie); 4926 } 4927 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz); 4928 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); 4929 if (ringp->s_ring_state & (S_RING_BLOCK | S_RING_PROC)) { 4930 /* 4931 * If ring is blocked due to lack of Tx 4932 * descs, just return. Worker thread 4933 * will get scheduled when Tx desc's 4934 * become available. 4935 */ 4936 mutex_exit(&ringp->s_ring_lock); 4937 return (cookie); 4938 } 4939 mac_soft_ring_worker_wakeup(ringp); 4940 mutex_exit(&ringp->s_ring_lock); 4941 return (cookie); 4942 } else { 4943 /* Default fanout mode */ 4944 /* 4945 * S_RING_BLOCKED is set when underlying NIC runs 4946 * out of Tx descs and messages start getting 4947 * queued. It won't get reset until 4948 * tx_srs_drain() completely drains out the 4949 * messages. 4950 */ 4951 mac_tx_stats_t stats; 4952 4953 if (ringp->s_ring_state & S_RING_ENQUEUED) { 4954 /* Tx descs/resources not available */ 4955 mutex_enter(&ringp->s_ring_lock); 4956 if (ringp->s_ring_state & S_RING_ENQUEUED) { 4957 cookie = mac_tx_sring_enqueue(ringp, mp_chain, 4958 flag, ret_mp); 4959 mutex_exit(&ringp->s_ring_lock); 4960 return (cookie); 4961 } 4962 /* 4963 * While we were computing mblk count, the 4964 * flow control condition got relieved. 4965 * Continue with the transmission. 4966 */ 4967 mutex_exit(&ringp->s_ring_lock); 4968 } 4969 4970 mp_chain = mac_tx_send(ringp->s_ring_tx_arg1, 4971 ringp->s_ring_tx_arg2, mp_chain, &stats); 4972 4973 /* 4974 * Multiple threads could be here sending packets. 4975 * Under such conditions, it is not possible to 4976 * automically set S_RING_BLOCKED bit to indicate 4977 * out of tx desc condition. To atomically set 4978 * this, we queue the returned packet and do 4979 * the setting of S_RING_BLOCKED in 4980 * mac_tx_soft_ring_drain(). 4981 */ 4982 if (mp_chain != NULL) { 4983 mutex_enter(&ringp->s_ring_lock); 4984 cookie = 4985 mac_tx_sring_enqueue(ringp, mp_chain, flag, ret_mp); 4986 mutex_exit(&ringp->s_ring_lock); 4987 return (cookie); 4988 } 4989 SRS_TX_STATS_UPDATE(mac_srs, &stats); 4990 SOFTRING_TX_STATS_UPDATE(ringp, &stats); 4991 4992 return (0); 4993 } 4994 } 4995