1 /*
2 * CDDL HEADER START
3 *
4 * The contents of this file are subject to the terms of the
5 * Common Development and Distribution License (the "License").
6 * You may not use this file except in compliance with the License.
7 *
8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
9 * or http://www.opensolaris.org/os/licensing.
10 * See the License for the specific language governing permissions
11 * and limitations under the License.
12 *
13 * When distributing Covered Code, include this CDDL HEADER in each
14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
15 * If applicable, add the following below this CDDL HEADER, with the
16 * fields enclosed by brackets "[]" replaced with your own identifying
17 * information: Portions Copyright [yyyy] [name of copyright owner]
18 *
19 * CDDL HEADER END
20 */
21 /*
22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved.
23 * Use is subject to license terms.
24 * Copyright 2018 Joyent, Inc.
25 * Copyright 2013 Nexenta Systems, Inc. All rights reserved.
26 * Copyright 2025 Oxide Computer Company
27 */
28
29 /*
30 * MAC data path
31 *
32 * The MAC data path is concerned with the flow of traffic from mac clients --
33 * DLS, IP, etc. -- to various GLDv3 device drivers -- e1000g, vnic, aggr,
34 * ixgbe, etc. -- and from the GLDv3 device drivers back to clients.
35 *
36 * -----------
37 * Terminology
38 * -----------
39 *
40 * MAC uses a lot of different, but related terms that are associated with the
41 * design and structure of the data path. Before we cover other aspects, first
42 * let's review the terminology that MAC uses.
43 *
44 * MAC
45 *
46 * This driver. It interfaces with device drivers and provides abstractions
47 * that the rest of the system consumes. All data links -- things managed
48 * with dladm(8), are accessed through MAC.
49 *
50 * GLDv3 DEVICE DRIVER
51 *
52 * A GLDv3 device driver refers to a driver, both for pseudo-devices and
53 * real devices, which implement the GLDv3 driver API. Common examples of
54 * these are igb and ixgbe, which are drivers for various Intel networking
55 * cards. These devices may or may not have various features, such as
56 * hardware rings and checksum offloading. For MAC, a GLDv3 device is the
57 * final point for the transmission of a packet and the starting point for
58 * the receipt of a packet.
59 *
60 * FLOWS
61 *
62 * At a high level, a flow refers to a series of packets that are related.
63 * Often times the term is used in the context of TCP to indicate a unique
64 * TCP connection and the traffic over it. However, a flow can exist at
65 * other levels of the system as well. MAC has a notion of a default flow
66 * which is used for all unicast traffic addressed to the address of a MAC
67 * device. For example, when a VNIC is created, a default flow is created
68 * for the VNIC's MAC address. In addition, flows are created for broadcast
69 * groups and a user may create a flow with flowadm(8).
70 *
71 * CLASSIFICATION
72 *
73 * Classification refers to the notion of identifying an incoming frame
74 * based on its destination address and optionally its source addresses and
75 * doing different processing based on that information. Classification can
76 * be done in both hardware and software. In general, we usually only
77 * classify based on the layer two destination, eg. for Ethernet, the
78 * destination MAC address.
79 *
80 * The system also will do classification based on layer three and layer
81 * four properties. This is used to support things like flowadm(8), which
82 * allows setting QoS and other properties on a per-flow basis.
83 *
84 * RING
85 *
86 * Conceptually, a ring represents a series of framed messages, often in a
87 * contiguous chunk of memory that acts as a circular buffer. Rings come in
88 * a couple of forms. Generally they are either a hardware construct (hw
89 * ring) or they are a software construct (sw ring) maintained by MAC.
90 *
91 * HW RING
92 *
93 * A hardware ring is a set of resources provided by a GLDv3 device driver
94 * (even if it is a pseudo-device). A hardware ring comes in two different
95 * forms: receive (rx) rings and transmit (tx) rings. An rx hw ring is
96 * something that has a unique DMA (direct memory access) region and
97 * generally supports some form of classification (though it isn't always
98 * used), as well as a means of generating an interrupt specific to that
99 * ring. For example, the device may generate a specific MSI-X for a PCI
100 * express device. A tx ring is similar, except that it is dedicated to
101 * transmission. It may also be a vector for enabling features such as VLAN
102 * tagging and large transmit offloading. It usually has its own dedicated
103 * interrupts for transmit being completed.
104 *
105 * SW RING
106 *
107 * A software ring is a construction of MAC. It represents the same thing
108 * that a hardware ring generally does, a collection of frames. However,
109 * instead of being in a contiguous ring of memory, they're instead linked
110 * by using the mblk_t's b_next pointer. Each frame may itself be multiple
111 * mblk_t's linked together by the b_cont pointer. A software ring always
112 * represents a collection of classified packets; however, it varies as to
113 * whether it uses only layer two information, or a combination of that and
114 * additional layer three and layer four data.
115 *
116 * FANOUT
117 *
118 * Fanout is the idea of spreading out the load of processing frames based
119 * on the source and destination information contained in the layer two,
120 * three, and four headers, such that the data can then be processed in
121 * parallel using multiple hardware threads.
122 *
123 * A fanout algorithm hashes the headers and uses that to place different
124 * flows into a bucket. The most important thing is that packets that are
125 * in the same flow end up in the same bucket. If they do not, performance
126 * can be adversely affected. Consider the case of TCP. TCP severely
127 * penalizes a connection if the data arrives out of order. If a given flow
128 * is processed on different CPUs, then the data will appear out of order,
129 * hence the invariant that fanout always hash a given flow to the same
130 * bucket and thus get processed on the same CPU.
131 *
132 * RECEIVE SIDE SCALING (RSS)
133 *
134 *
135 * Receive side scaling is a term that isn't common in illumos, but is used
136 * by vendors and was popularized by Microsoft. It refers to the idea of
137 * spreading the incoming receive load out across multiple interrupts which
138 * can be directed to different CPUs. This allows a device to leverage
139 * hardware rings even when it doesn't support hardware classification. The
140 * hardware uses an algorithm to perform fanout that ensures the flow
141 * invariant is maintained.
142 *
143 * SOFT RING SET
144 *
145 * A soft ring set, commonly abbreviated SRS, is a collection of rings and
146 * is used for both transmitting and receiving. It is maintained in the
147 * structure mac_soft_ring_set_t. A soft ring set is usually associated
148 * with flows, and coordinates both the use of hardware and software rings.
149 * Because the use of hardware rings can change as devices such as VNICs
150 * come and go, we always ensure that the set has software classification
151 * rules that correspond to the hardware classification rules from rings.
152 *
153 * Soft ring sets are also used for the enforcement of various QoS
154 * properties. For example, if a bandwidth limit has been placed on a
155 * specific flow or device, then that will be enforced by the soft ring
156 * set.
157 *
158 * SERVICE ATTACHMENT POINT (SAP)
159 *
160 * The service attachment point is a DLPI (Data Link Provider Interface)
161 * concept; however, it comes up quite often in MAC. Most MAC devices speak
162 * a protocol that has some notion of different channels or message type
163 * identifiers. For example, Ethernet defines an EtherType which is a part
164 * of the Ethernet header and defines the particular protocol of the data
165 * payload. If the EtherType is set to 0x0800, then it defines that the
166 * contents of that Ethernet frame is IPv4 traffic. For Ethernet, the
167 * EtherType is the SAP.
168 *
169 * In DLPI, a given consumer attaches to a specific SAP. In illumos, the ip
170 * and arp drivers attach to the EtherTypes for IPv4, IPv6, and ARP. Using
171 * libdlpi(3LIB) user software can attach to arbitrary SAPs. With the
172 * exception of 802.1Q VLAN tagged traffic, MAC itself does not directly
173 * consume the SAP; however, it uses that information as part of hashing
174 * and it may be used as part of the construction of flows.
175 *
176 * PRIMARY MAC CLIENT
177 *
178 * The primary mac client refers to a mac client whose unicast address
179 * matches the address of the device itself. For example, if the system has
180 * instance of the e1000g driver such as e1000g0, e1000g1, etc., the
181 * primary mac client is the one named after the device itself. VNICs that
182 * are created on top of such devices are not the primary client.
183 *
184 * TRANSMIT DESCRIPTORS
185 *
186 * Transmit descriptors are a resource that most GLDv3 device drivers have.
187 * Generally, a GLDv3 device driver takes a frame that's meant to be output
188 * and puts a copy of it into a region of memory. Each region of memory
189 * usually has an associated descriptor that the device uses to manage
190 * properties of the frames. Devices have a limited number of such
191 * descriptors. They get reclaimed once the device finishes putting the
192 * frame on the wire.
193 *
194 * If the driver runs out of transmit descriptors, for example, the OS is
195 * generating more frames than it can put on the wire, then it will return
196 * them back to the MAC layer.
197 *
198 * ---------------------------------
199 * Rings, Classification, and Fanout
200 * ---------------------------------
201 *
202 * The heart of MAC is made up of rings, and not those that Elven-kings wear.
203 * When receiving a packet, MAC breaks the work into two different, though
204 * interrelated phases. The first phase is generally classification and then the
205 * second phase is generally fanout. When a frame comes in from a GLDv3 Device,
206 * MAC needs to determine where that frame should be delivered. If it's a
207 * unicast frame (say a normal TCP/IP packet), then it will be delivered to a
208 * single MAC client; however, if it's a broadcast or multicast frame, then MAC
209 * may need to deliver it to multiple MAC clients.
210 *
211 * On transmit, classification isn't quite as important, but may still be used.
212 * Unlike with the receive path, the classification is not used to determine
213 * devices that should transmit something, but rather is used for special
214 * properties of a flow, eg. bandwidth limits for a given IP address, device, or
215 * connection.
216 *
217 * MAC employs a software classifier and leverages hardware classification as
218 * well. The software classifier can leverage the full layer two information,
219 * source, destination, VLAN, and SAP. If the SAP indicates that IP traffic is
220 * being sent, it can classify based on the IP header, and finally, it also
221 * knows how to classify based on the local and remote ports of TCP, UDP, and
222 * SCTP.
223 *
224 * Hardware classifiers vary in capability. Generally all hardware classifiers
225 * provide the capability to classify based on the destination MAC address. Some
226 * hardware has additional filters built in for performing more in-depth
227 * classification; however, it often has much more limited resources for these
228 * activities as compared to the layer two destination address classification.
229 *
230 * The modus operandi in MAC is to always ensure that we have software-based
231 * capabilities and rules in place and then to supplement that with hardware
232 * resources when available. In general, simple layer two classification is
233 * sufficient and nothing else is used, unless a specific flow is created with
234 * tools such as flowadm(8) or bandwidth limits are set on a device with
235 * dladm(8).
236 *
237 * RINGS AND GROUPS
238 *
239 * To get into how rings and classification play together, it's first important
240 * to understand how hardware devices commonly associate rings and allow them to
241 * be programmed. Recall that a hardware ring should be thought of as a DMA
242 * buffer and an interrupt resource. Rings are then collected into groups. A
243 * group itself has a series of classification rules. One or more MAC addresses
244 * are assigned to a group.
245 *
246 * Hardware devices vary in terms of what capabilities they provide. Sometimes
247 * they allow for a dynamic assignment of rings to a group and sometimes they
248 * have a static assignment of rings to a group. For example, the ixgbe driver
249 * has a static assignment of rings to groups such that every group has exactly
250 * one ring and the number of groups is equal to the number of rings.
251 *
252 * Classification and receive side scaling both come into play with how a device
253 * advertises itself to MAC and how MAC uses it. If a device supports layer two
254 * classification of frames, then MAC will assign MAC addresses to a group as a
255 * form of primary classification. If a single MAC address is assigned to a
256 * group, a common case, then MAC will consider packets that come in from rings
257 * on that group to be fully classified and will not need to do any software
258 * classification unless a specific flow has been created.
259 *
260 * If a device supports receive side scaling, then it may advertise or support
261 * groups with multiple rings. In those cases, then receive side scaling will
262 * come into play and MAC will use that as a means of fanning out received
263 * frames across multiple CPUs. This can also be combined with groups that
264 * support layer two classification.
265 *
266 * If a device supports dynamic assignments of rings to groups, then MAC will
267 * change around the way that rings are assigned to various groups as devices
268 * come and go from the system. For example, when a VNIC is created, a new flow
269 * will be created for the VNIC's MAC address. If a hardware ring is available,
270 * MAC may opt to reassign it from one group to another.
271 *
272 * ASSIGNMENT OF HARDWARE RINGS
273 *
274 * This is a bit of a complicated subject that varies depending on the device,
275 * the use of aggregations, the special nature of the primary mac client. This
276 * section deserves being fleshed out.
277 *
278 * FANOUT
279 *
280 * illumos uses fanout to help spread out the incoming processing load of chains
281 * of frames away from a single CPU. If a device supports receive side scaling,
282 * then that provides an initial form of fanout; however, what we're concerned
283 * with all happens after the context of a given set of frames being classified
284 * to a soft ring set.
285 *
286 * After frames reach a soft ring set and account for any potential bandwidth
287 * related accounting, they may be fanned out based on one of the following
288 * three modes:
289 *
290 * o No Fanout
291 * o Protocol level fanout
292 * o Full software ring protocol fanout
293 *
294 * MAC makes the determination as to which of these modes a given soft ring set
295 * obtains based on parameters such as whether or not it's the primary mac
296 * client, whether it's on a 10 GbE or faster device, user controlled dladm(8)
297 * properties, and the nature of the hardware and the resources that it has.
298 *
299 * When there is no fanout, MAC does not create any soft rings for a device and
300 * the device has frames delivered directly to the MAC client.
301 *
302 * Otherwise, all fanout is performed by software. MAC divides incoming frames
303 * into one of five buckets -- IPv4 TCP traffic, IPv4 UDP traffic, IPv6 TCP
304 * traffic, IPv6 UDP traffic, and everything else. Regardless of the type of
305 * fanout, these five categories of buckets are always used.
306 *
307 * The difference between protocol level fanout and full software ring protocol
308 * fanout is the number of software rings that end up getting created. The
309 * system always uses the same number of software rings per protocol bucket. So
310 * in the first case when we're just doing protocol level fanout, we just create
311 * one software ring each for IPv4 TCP traffic, IPv4 UDP traffic, IPv6 TCP
312 * traffic, IPv6 UDP traffic, and everything else.
313 *
314 * In the case where we do full software ring protocol fanout, we generally use
315 * mac_compute_soft_ring_count() to determine the number of rings. There are
316 * other combinations of properties and devices that may send us down other
317 * paths, but this is a common starting point. If it's a non-bandwidth enforced
318 * device and we're on at least a 10 GbE link, then we'll use eight soft rings
319 * per protocol bucket as a starting point. See mac_compute_soft_ring_count()
320 * for more information on the total number.
321 *
322 * For each of these rings, we create a mac_soft_ring_t and an associated worker
323 * thread. Particularly when doing full software ring protocol fanout, we bind
324 * each of the worker threads to individual CPUs.
325 *
326 * The other advantage of these software rings is that it allows upper layers to
327 * optionally poll on them. For example, TCP can leverage an squeue to poll on
328 * the software ring, see squeue.c for more information.
329 *
330 * DLS BYPASS
331 *
332 * DLS is the data link services module. It interfaces with DLPI, which is the
333 * primary way that other parts of the system such as IP interface with the MAC
334 * layer. While DLS is traditionally a STREAMS-based interface, it allows for
335 * certain modules such as IP to negotiate various more modern interfaces to be
336 * used, which are useful for higher performance and allow it to use direct
337 * function calls to DLS instead of using STREAMS.
338 *
339 * When we have TCP or UDP software rings, then traffic on those rings is
340 * eligible for what we call the dls bypass. In those cases, rather than going
341 * out mac_rx_deliver() to DLS, DLS instead registers them to go directly via
342 * the direct callback registered with DLS, generally ip_input().
343 *
344 * HARDWARE RING POLLING
345 *
346 * GLDv3 devices with hardware rings generally deliver chains of messages
347 * (mblk_t chain) during the context of a single interrupt. However, interrupts
348 * are not the only way that these devices may be used. As part of implementing
349 * ring support, a GLDv3 device driver must have a way to disable the generation
350 * of that interrupt and allow for the operating system to poll on that ring.
351 *
352 * To implement this, every soft ring set has a worker thread and a polling
353 * thread. If a sufficient packet rate comes into the system, MAC will 'blank'
354 * (disable) interrupts on that specific ring and the polling thread will start
355 * consuming packets from the hardware device and deliver them to the soft ring
356 * set, where the worker thread will take over.
357 *
358 * Once the rate of packet intake drops down below a certain threshold, then
359 * polling on the hardware ring will be quiesced and interrupts will be
360 * re-enabled for the given ring. This effectively allows the system to shift
361 * how it handles a ring based on its load. At high packet rates, polling on the
362 * device as opposed to relying on interrupts can actually reduce overall system
363 * load due to the minimization of interrupt activity.
364 *
365 * Note the importance of each ring having its own interrupt source. The whole
366 * idea here is that we do not disable interrupts on the device as a whole, but
367 * rather each ring can be independently toggled.
368 *
369 * USE OF WORKER THREADS
370 *
371 * Both the soft ring set and individual soft rings have a worker thread
372 * associated with them that may be bound to a specific CPU in the system. Any
373 * such assignment will get reassessed as part of dynamic reconfiguration events
374 * in the system such as the onlining and offlining of CPUs and the creation of
375 * CPU partitions.
376 *
377 * In many cases, while in an interrupt, we try to deliver a frame all the way
378 * through the stack in the context of the interrupt itself. However, if the
379 * amount of queued frames has exceeded a threshold, then we instead defer to
380 * the worker thread to do this work and signal it. This is particularly useful
381 * when you have the soft ring set delivering frames into multiple software
382 * rings. If it was only delivering frames into a single software ring then
383 * there'd be no need to have another thread take over. However, if it's
384 * delivering chains of frames to multiple rings, then it's worthwhile to have
385 * the worker for the software ring take over so that the different software
386 * rings can be processed in parallel.
387 *
388 * In a similar fashion to the hardware polling thread, if we don't have a
389 * backlog or there's nothing to do, then the worker thread will go back to
390 * sleep and frames can be delivered all the way from an interrupt. This
391 * behavior is useful as it's designed to minimize latency and the default
392 * disposition of MAC is to optimize for latency.
393 *
394 * MAINTAINING CHAINS
395 *
396 * Another useful idea that MAC uses is to try and maintain frames in chains for
397 * as long as possible. The idea is that all of MAC can handle chains of frames
398 * structured as a series of mblk_t structures linked with the b_next pointer.
399 * When performing software classification and software fanout, MAC does not
400 * simply determine the destination and send the frame along. Instead, in the
401 * case of classification, it tries to maintain a chain for as long as possible
402 * before passing it along and performing additional processing.
403 *
404 * In the case of fanout, MAC first determines what the target software ring is
405 * for every frame in the original chain and constructs a new chain for each
406 * target. MAC then delivers the new chain to each software ring in succession.
407 *
408 * The whole rationale for doing this is that we want to try and maintain the
409 * pipe as much as possible and deliver as many frames through the stack at once
410 * that we can, rather than just pushing a single frame through. This can often
411 * help bring down latency and allows MAC to get a better sense of the overall
412 * activity in the system and properly engage worker threads.
413 *
414 * --------------------
415 * Bandwidth Management
416 * --------------------
417 *
418 * Bandwidth management is something that's built into the soft ring set itself.
419 * When bandwidth limits are placed on a flow, a corresponding soft ring set is
420 * toggled into bandwidth mode. This changes how we transmit and receive the
421 * frames in question.
422 *
423 * Bandwidth management is done on a per-tick basis. We translate the user's
424 * requested bandwidth from a quantity per-second into a quantity per-tick. MAC
425 * cannot process a frame across more than one tick, thus it sets a lower bound
426 * for the bandwidth cap to be a single MTU. This also means that when
427 * hires ticks are enabled (hz is set to 1000), that the minimum amount of
428 * bandwidth is higher, because the number of ticks has increased and MAC has to
429 * go from accepting 100 packets / sec to 1000 / sec.
430 *
431 * The bandwidth counter is reset by either the soft ring set's worker thread or
432 * a thread that is doing an inline transmit or receive if they discover that
433 * the current tick is in the future from the recorded tick.
434 *
435 * Whenever we're receiving or transmitting data, we end up leaving most of the
436 * work to the soft ring set's worker thread. This forces data inserted into the
437 * soft ring set to be effectively serialized and allows us to exhume bandwidth
438 * at a reasonable rate. If there is nothing in the soft ring set at the moment
439 * and the set has available bandwidth, then it may processed inline.
440 * Otherwise, the worker is responsible for taking care of the soft ring set.
441 *
442 * ---------------------
443 * The Receive Data Path
444 * ---------------------
445 *
446 * The following series of ASCII art images breaks apart the way that a frame
447 * comes in and is processed in MAC.
448 *
449 * Part 1 -- Initial frame receipt, SRS classification
450 *
451 * Here, a frame is received by a GLDv3 driver, generally in the context of an
452 * interrupt, and it ends up in mac_rx_common(). A driver calls either mac_rx or
453 * mac_rx_ring, depending on whether or not it supports rings and can identify
454 * the interrupt as having come from a specific ring. Here we determine whether
455 * or not it's fully classified and perform software classification as
456 * appropriate. From here, everything always ends up going to either entry [A]
457 * or entry [B] based on whether or not they have subflow processing needed. We
458 * leave via fanout or delivery.
459 *
460 * +===========+
461 * v hardware v
462 * v interrupt v
463 * +===========+
464 * |
465 * * . . appropriate
466 * | upcall made
467 * | by GLDv3 driver . . always
468 * | .
469 * +--------+ | +----------+ . +---------------+
470 * | GLDv3 | +---->| mac_rx |-----*--->| mac_rx_common |
471 * | Driver |-->--+ +----------+ +---------------+
472 * +--------+ | ^ |
473 * | | ^ v
474 * ^ | * . . always +----------------------+
475 * | | | | mac_promisc_dispatch |
476 * | | +-------------+ +----------------------+
477 * | +--->| mac_rx_ring | |
478 * | +-------------+ * . . hw classified
479 * | v or single flow?
480 * | |
481 * | +--------++--------------+
482 * | | | * hw class,
483 * | | * hw classified | subflows
484 * | no hw class and . * | or single | exist
485 * | subflows | | flow |
486 * | | v v
487 * | | +-----------+ +-----------+
488 * | | | goto | | goto |
489 * | | | entry [A] | | entry [B] |
490 * | | +-----------+ +-----------+
491 * | v ^
492 * | +-------------+ |
493 * | | mac_rx_flow | * SRS and flow found,
494 * | +-------------+ | call flow cb
495 * | | +------+
496 * | v |
497 * v +==========+ +-----------------+
498 * | v For each v--->| mac_rx_classify |
499 * +----------+ v mblk_t v +-----------------+
500 * | srs | +==========+
501 * | pollling |
502 * | thread |->------------------------------------------+
503 * +----------+ |
504 * v . inline
505 * +--------------------+ +----------+ +---------+ .
506 * [A]---->| mac_rx_srs_process |-->| check bw |-->| enqueue |--*---------+
507 * +--------------------+ | limits | | frames | |
508 * ^ +----------+ | to SRS | |
509 * | +---------+ |
510 * | send chain +--------+ | |
511 * * when clasified | signal | * BW limits, |
512 * | flow changes | srs |<---+ loopback, |
513 * | | worker | stack too |
514 * | +--------+ deep |
515 * +-----------------+ +--------+ |
516 * | mac_flow_lookup | | srs | +---------------------+ |
517 * +-----------------+ | worker |---->| mac_rx_srs_drain |<---+
518 * ^ | thread | | mac_rx_srs_drain_bw |
519 * | +--------+ +---------------------+
520 * | |
521 * +----------------------------+ * software rings
522 * [B]-->| mac_rx_srs_subflow_process | | for fanout?
523 * +----------------------------+ |
524 * +----------+-----------+
525 * | |
526 * v v
527 * +--------+ +--------+
528 * | goto | | goto |
529 * | Part 2 | | Part 3 |
530 * +--------+ +--------+
531 *
532 * Part 2 -- Fanout
533 *
534 * This part is concerned with using software fanout to assign frames to
535 * software rings and then deliver them to MAC clients or allow those rings to
536 * be polled upon. While there are two different primary fanout entry points,
537 * mac_rx_fanout and mac_rx_proto_fanout, they behave in similar ways, and aside
538 * from some of the individual hashing techniques used, most of the general
539 * flow is the same.
540 *
541 * +--------+ +-------------------+
542 * | From |---+--------->| mac_rx_srs_fanout |----+
543 * | Part 1 | | +-------------------+ | +=================+
544 * +--------+ | | v for each mblk_t v
545 * * . . protocol only +--->v assign to new v
546 * | fanout | v chain based on v
547 * | | v hash % nrings v
548 * | +-------------------------+ | +=================+
549 * +--->| mac_rx_srs_proto_fanout |----+ |
550 * +-------------------------+ |
551 * v
552 * +------------+ +--------------------------+ +================+
553 * | enqueue in |<---| mac_rx_soft_ring_process |<------v for each chain v
554 * | soft ring | +--------------------------+ +================+
555 * +------------+
556 * | +-----------+
557 * * soft ring set | soft ring |
558 * | empty and no | worker |
559 * | worker? | thread |
560 * | +-----------+
561 * +------*----------------+ |
562 * | . | v
563 * No . * . Yes | +------------------------+
564 * | +----<--| mac_rx_soft_ring_drain |
565 * | | +------------------------+
566 * v |
567 * +-----------+ v
568 * | signal | +---------------+
569 * | soft ring | | Deliver chain |
570 * | worker | | goto Part 3 |
571 * +-----------+ +---------------+
572 *
573 *
574 * Part 3 -- Packet Delivery
575 *
576 * Here, we go through and deliver the mblk_t chain directly to a given
577 * processing function. In a lot of cases this is mac_rx_deliver(). In the case
578 * of DLS bypass being used, then instead we end up going ahead and deliver it
579 * to the direct callback registered with DLS, generally ip_input.
580 *
581 *
582 * +---------+ +----------------+ +------------------+
583 * | From |---+------->| mac_rx_deliver |--->| Off to DLS, or |
584 * | Parts 1 | | +----------------+ | other MAC client |
585 * | and 2 | * DLS bypass +------------------+
586 * +---------+ | enabled +----------+ +-------------+
587 * +---------->| ip_input |--->| To IP |
588 * +----------+ | and beyond! |
589 * +-------------+
590 *
591 * ----------------------
592 * The Transmit Data Path
593 * ----------------------
594 *
595 * Before we go into the images, it's worth talking about a problem that is a
596 * bit different from the receive data path. GLDv3 device drivers have a finite
597 * amount of transmit descriptors. When they run out, they return unused frames
598 * back to MAC. MAC, at this point has several options about what it will do,
599 * which vary based upon the settings that the client uses.
600 *
601 * When a device runs out of descriptors, the next thing that MAC does is
602 * enqueue them off of the soft ring set or a software ring, depending on the
603 * configuration of the soft ring set. MAC will enqueue up to a high watermark
604 * of mblk_t chains, at which point it will indicate flow control back to the
605 * client. Once this condition is reached, any mblk_t chains that were not
606 * enqueued will be returned to the caller and they will have to decide what to
607 * do with them. There are various flags that control this behavior that a
608 * client may pass, which are discussed below.
609 *
610 * When this condition is hit, MAC also returns a cookie to the client in
611 * addition to unconsumed frames. Clients can poll on that cookie and register a
612 * callback with MAC to be notified when they are no longer subject to flow
613 * control, at which point they may continue to call mac_tx(). This flow control
614 * actually manages to work itself all the way up the stack, back through dls,
615 * to ip, through the various protocols, and to sockfs.
616 *
617 * While the behavior described above is the default, this behavior can be
618 * modified. There are two alternate modes, described below, which are
619 * controlled with flags.
620 *
621 * DROP MODE
622 *
623 * This mode is controlled by having the client pass the MAC_DROP_ON_NO_DESC
624 * flag. When this is passed, if a device driver runs out of transmit
625 * descriptors, then the MAC layer will drop any unsent traffic. The client in
626 * this case will never have any frames returned to it.
627 *
628 * DON'T ENQUEUE
629 *
630 * This mode is controlled by having the client pass the MAC_TX_NO_ENQUEUE flag.
631 * If the MAC_DROP_ON_NO_DESC flag is also passed, it takes precedence. In this
632 * mode, when we hit a case where a driver runs out of transmit descriptors,
633 * then instead of enqueuing packets in a soft ring set or software ring, we
634 * instead return the mblk_t chain back to the caller and immediately put the
635 * soft ring set into flow control mode.
636 *
637 * The following series of ASCII art images describe the transmit data path that
638 * MAC clients enter into based on calling into mac_tx(). A soft ring set has a
639 * transmission function associated with it. There are seven possible
640 * transmission modes, some of which share function entry points. The one that a
641 * soft ring set gets depends on properties such as whether there are
642 * transmission rings for fanout, whether the device involves aggregations,
643 * whether any bandwidth limits exist, etc.
644 *
645 *
646 * Part 1 -- Initial checks
647 *
648 * * . called by
649 * | MAC clients
650 * v . . No
651 * +--------+ +-----------+ . +-------------------+ +====================+
652 * | mac_tx |->| device |-*-->| mac_protect_check |->v Is this the simple v
653 * +--------+ | quiesced? | +-------------------+ v case? See [1] v
654 * +-----------+ | +====================+
655 * * . Yes * failed |
656 * v | frames |
657 * +--------------+ | +-------+---------+
658 * | freemsgchain |<---------+ Yes . * No . *
659 * +--------------+ v v
660 * +-----------+ +--------+
661 * | goto | | goto |
662 * | Part 2 | | SRS TX |
663 * | Entry [A] | | func |
664 * +-----------+ +--------+
665 * | |
666 * | v
667 * | +--------+
668 * +---------->| return |
669 * | cookie |
670 * +--------+
671 *
672 * [1] The simple case refers to the SRS being configured with the
673 * SRS_TX_DEFAULT transmission mode, having a single mblk_t (not a chain), their
674 * being only a single active client, and not having a backlog in the srs.
675 *
676 *
677 * Part 2 -- The SRS transmission functions
678 *
679 * This part is a bit more complicated. The different transmission paths often
680 * leverage one another. In this case, we'll draw out the more common ones
681 * before the parts that depend upon them. Here, we're going to start with the
682 * workings of mac_tx_send() a common function that most of the others end up
683 * calling.
684 *
685 * +-------------+
686 * | mac_tx_send |
687 * +-------------+
688 * |
689 * v
690 * +=============+ +==============+
691 * v more than v--->v check v
692 * v one client? v v VLAN and add v
693 * +=============+ v VLAN tags v
694 * | +==============+
695 * | |
696 * +------------------+
697 * |
698 * | [A]
699 * v |
700 * +============+ . No v
701 * v more than v . +==========+ +--------------------------+
702 * v one active v-*---->v for each v---->| mac_promisc_dispatch_one |---+
703 * v client? v v mblk_t v +--------------------------+ |
704 * +============+ +==========+ ^ |
705 * | | +==========+ |
706 * * . Yes | v hardware v<-------+
707 * v +------------+ v rings? v
708 * +==========+ | +==========+
709 * v for each v No . . . * |
710 * v mblk_t v specific | |
711 * +==========+ flow | +-----+-----+
712 * | | | |
713 * v | v v
714 * +-----------------+ | +-------+ +---------+
715 * | mac_tx_classify |------------+ | GLDv3 | | GLDv3 |
716 * +-----------------+ |TX func| | ring tx |
717 * | +-------+ | func |
718 * * Specific flow, generally | +---------+
719 * | bcast, mcast, loopback | |
720 * v +-----+-----+
721 * +==========+ +---------+ |
722 * v valid L2 v--*--->| freemsg | v
723 * v header v . No +---------+ +-------------------+
724 * +==========+ | return unconsumed |
725 * * . Yes | frames to the |
726 * v | caller |
727 * +===========+ +-------------------+
728 * v braodcast v +----------------+ ^
729 * v flow? v--*-->| mac_bcast_send |------------------+
730 * +===========+ . +----------------+ |
731 * | . . Yes |
732 * No . * v
733 * | +---------------------+ +---------------+ +----------+
734 * +->|mac_promisc_dispatch |->| mac_fix_cksum |->| flow |
735 * +---------------------+ +---------------+ | callback |
736 * +----------+
737 *
738 *
739 * In addition, many but not all of the routines, all rely on
740 * mac_tx_softring_process as an entry point.
741 *
742 *
743 * . No . No
744 * +--------------------------+ +========+ . +===========+ . +-------------+
745 * | mac_tx_soft_ring_process |-->v worker v-*->v out of tx v-*->| goto |
746 * +--------------------------+ v only? v v descr.? v | mac_tx_send |
747 * +========+ +===========+ +-------------+
748 * Yes . * * . Yes |
749 * . No v | v
750 * v=========+ . +===========+ . Yes | Yes . +==========+
751 * v apppend v<--*----------v out of tx v-*-------+---------*--v returned v
752 * v mblk_t v v descr.? v | v frames? v
753 * v chain v +===========+ | +==========+
754 * +=========+ | *. No
755 * | | v
756 * v v +------------+
757 * +===================+ +----------------------+ | done |
758 * v worker scheduled? v | mac_tx_sring_enqueue | | processing |
759 * v Out of tx descr? v +----------------------+ +------------+
760 * +===================+ |
761 * | | . Yes v
762 * * Yes * No . +============+
763 * | v +-*---------v drop on no v
764 * | +========+ v v TX desc? v
765 * | v wake v +----------+ +============+
766 * | v worker v | mac_pkt_ | * . No
767 * | +========+ | drop | | . Yes . No
768 * | | +----------+ v . .
769 * | | v ^ +===============+ . +========+ .
770 * +--+--------+---------+ | v Don't enqueue v-*->v ring v-*----+
771 * | | v Set? v v empty? v |
772 * | +---------------+ +===============+ +========+ |
773 * | | | | |
774 * | | +-------------------+ | |
775 * | *. Yes | +---------+ |
776 * | | v v v
777 * | | +===========+ +========+ +--------------+
778 * | +<-v At hiwat? v v append v | return |
779 * | +===========+ v mblk_t v | mblk_t chain |
780 * | * No v chain v | and flow |
781 * | v +========+ | control |
782 * | +=========+ | | cookie |
783 * | v append v v +--------------+
784 * | v mblk_t v +========+
785 * | v chain v v wake v +------------+
786 * | +=========+ v worker v-->| done |
787 * | | +========+ | processing |
788 * | v .. Yes +------------+
789 * | +=========+ . +========+
790 * | v first v--*-->v wake v
791 * | v append? v v worker v
792 * | +=========+ +========+
793 * | | |
794 * | No . * |
795 * | v |
796 * | +--------------+ |
797 * +------>| Return | |
798 * | flow control |<------------+
799 * | cookie |
800 * +--------------+
801 *
802 *
803 * The remaining images are all specific to each of the different transmission
804 * modes.
805 *
806 * SRS TX DEFAULT
807 *
808 * [ From Part 1 ]
809 * |
810 * v
811 * +-------------------------+
812 * | mac_tx_single_ring_mode |
813 * +-------------------------+
814 * |
815 * | . Yes
816 * v .
817 * +==========+ . +============+
818 * v SRS v-*->v Try to v---->---------------------+
819 * v backlog? v v enqueue in v |
820 * +==========+ v SRS v-->------+ * . . Queue too
821 * | +============+ * don't enqueue | deep or
822 * * . No ^ | | flag or at | drop flag
823 * | | v | hiwat, |
824 * v | | | return +---------+
825 * +-------------+ | | | cookie | freemsg |
826 * | goto |-*-----+ | | +---------+
827 * | mac_tx_send | . returned | | |
828 * +-------------+ mblk_t | | |
829 * | | | |
830 * | | | |
831 * * . . all mblk_t * queued, | |
832 * v consumed | may return | |
833 * +-------------+ | tx cookie | |
834 * | SRS TX func |<------------+------------+----------------+
835 * | completed |
836 * +-------------+
837 *
838 * SRS_TX_SERIALIZE
839 *
840 * +------------------------+
841 * | mac_tx_serializer_mode |
842 * +------------------------+
843 * |
844 * | . No
845 * v .
846 * +============+ . +============+ +-------------+ +============+
847 * v srs being v-*->v set SRS v--->| goto |-->v remove SRS v
848 * v processed? v v proc flags v | mac_tx_send | v proc flag v
849 * +============+ +============+ +-------------+ +============+
850 * | |
851 * * Yes |
852 * v . No v
853 * +--------------------+ . +==========+
854 * | mac_tx_srs_enqueue | +------------------------*-----<--v returned v
855 * +--------------------+ | v frames? v
856 * | | . Yes +==========+
857 * | | . |
858 * | | . +=========+ v
859 * v +-<-*-v queued v +--------------------+
860 * +-------------+ | v frames? v<----| mac_tx_srs_enqueue |
861 * | SRS TX func | | +=========+ +--------------------+
862 * | completed, |<------+ * . Yes
863 * | may return | | v
864 * | cookie | | +========+
865 * +-------------+ +-<---v wake v
866 * v worker v
867 * +========+
868 *
869 *
870 * SRS_TX_FANOUT
871 *
872 * . Yes
873 * +--------------------+ +=============+ . +--------------------------+
874 * | mac_tx_fanout_mode |--->v Have fanout v-*-->| goto |
875 * +--------------------+ v hint? v | mac_rx_soft_ring_process |
876 * +=============+ +--------------------------+
877 * * . No |
878 * v ^
879 * +===========+ |
880 * +--->v for each v +===============+
881 * | v mblk_t v v pick softring v
882 * same * +===========+ v from hash v
883 * hash | | +===============+
884 * | v |
885 * | +--------------+ |
886 * +---| mac_pkt_hash |--->*------------+
887 * +--------------+ . different
888 * hash or
889 * done proc.
890 * SRS_TX_AGGR chain
891 *
892 * +------------------+ +================================+
893 * | mac_tx_aggr_mode |--->v Use aggr capab function to v
894 * +------------------+ v find appropriate tx ring. v
895 * v Applies hash based on aggr v
896 * v policy, see mac_tx_aggr_mode() v
897 * +================================+
898 * |
899 * v
900 * +-------------------------------+
901 * | goto |
902 * | mac_rx_srs_soft_ring_process |
903 * +-------------------------------+
904 *
905 *
906 * SRS_TX_BW, SRS_TX_BW_FANOUT, SRS_TX_BW_AGGR
907 *
908 * Note, all three of these tx functions start from the same place --
909 * mac_tx_bw_mode().
910 *
911 * +----------------+
912 * | mac_tx_bw_mode |
913 * +----------------+
914 * |
915 * v . No . No . Yes
916 * +==============+ . +============+ . +=============+ . +=========+
917 * v Out of BW? v--*->v SRS empty? v--*->v reset BW v-*->v Bump BW v
918 * +==============+ +============+ v tick count? v v Usage v
919 * | | +=============+ +=========+
920 * | +---------+ | |
921 * | | +--------------------+ |
922 * | | | +----------------------+
923 * v | v v
924 * +===============+ | +==========+ +==========+ +------------------+
925 * v Don't enqueue v | v set bw v v Is aggr? v--*-->| goto |
926 * v flag set? v | v enforced v +==========+ . | mac_tx_aggr_mode |-+
927 * +===============+ | +==========+ | . +------------------+ |
928 * | Yes .* | | No . * . |
929 * | | | | | . Yes |
930 * * . No | | v | |
931 * | +---------+ | +========+ v +======+ |
932 * | | freemsg | | v append v +============+ . Yes v pick v |
933 * | +---------+ | v mblk_t v v Is fanout? v--*---->v ring v |
934 * | | | v chain v +============+ +======+ |
935 * +------+ | +========+ | | |
936 * v | | v v |
937 * +---------+ | v +-------------+ +--------------------+ |
938 * | return | | +========+ | goto | | goto | |
939 * | flow | | v wakeup v | mac_tx_send | | mac_tx_fanout_mode | |
940 * | control | | v worker v +-------------+ +--------------------+ |
941 * | cookie | | +========+ | | |
942 * +---------+ | | | +------+------+
943 * | v | |
944 * | +---------+ | v
945 * | | return | +============+ +------------+
946 * | | flow | v unconsumed v-------+ | done |
947 * | | control | v frames? v | | processing |
948 * | | cookie | +============+ | +------------+
949 * | +---------+ | |
950 * | Yes * |
951 * | | |
952 * | +===========+ |
953 * | v subtract v |
954 * | v unused bw v |
955 * | +===========+ |
956 * | | |
957 * | v |
958 * | +--------------------+ |
959 * +------------->| mac_tx_srs_enqueue | |
960 * +--------------------+ |
961 * | |
962 * | |
963 * +------------+ |
964 * | return fc | |
965 * | cookie and |<------+
966 * | mblk_t |
967 * +------------+
968 */
969
970 #include <sys/types.h>
971 #include <sys/callb.h>
972 #include <sys/pattr.h>
973 #include <sys/sdt.h>
974 #include <sys/strsubr.h>
975 #include <sys/strsun.h>
976 #include <sys/vlan.h>
977 #include <sys/stack.h>
978 #include <sys/archsystm.h>
979 #include <inet/ipsec_impl.h>
980 #include <inet/ip_impl.h>
981 #include <inet/sadb.h>
982 #include <inet/ipsecesp.h>
983 #include <inet/ipsecah.h>
984 #include <inet/ip6.h>
985
986 #include <sys/mac_impl.h>
987 #include <sys/mac_client_impl.h>
988 #include <sys/mac_client_priv.h>
989 #include <sys/mac_soft_ring.h>
990 #include <sys/mac_flow_impl.h>
991
992 static mac_tx_cookie_t mac_tx_single_ring_mode(mac_soft_ring_set_t *, mblk_t *,
993 uintptr_t, uint16_t, mblk_t **);
994 static mac_tx_cookie_t mac_tx_serializer_mode(mac_soft_ring_set_t *, mblk_t *,
995 uintptr_t, uint16_t, mblk_t **);
996 static mac_tx_cookie_t mac_tx_fanout_mode(mac_soft_ring_set_t *, mblk_t *,
997 uintptr_t, uint16_t, mblk_t **);
998 static mac_tx_cookie_t mac_tx_bw_mode(mac_soft_ring_set_t *, mblk_t *,
999 uintptr_t, uint16_t, mblk_t **);
1000 static mac_tx_cookie_t mac_tx_aggr_mode(mac_soft_ring_set_t *, mblk_t *,
1001 uintptr_t, uint16_t, mblk_t **);
1002
1003 typedef struct mac_tx_mode_s {
1004 mac_tx_srs_mode_t mac_tx_mode;
1005 mac_tx_func_t mac_tx_func;
1006 } mac_tx_mode_t;
1007
1008 /*
1009 * There are seven modes of operation on the Tx side. These modes get set
1010 * in mac_tx_srs_setup(). Except for the experimental TX_SERIALIZE mode,
1011 * none of the other modes are user configurable. They get selected by
1012 * the system depending upon whether the link (or flow) has multiple Tx
1013 * rings or a bandwidth configured, or if the link is an aggr, etc.
1014 *
1015 * When the Tx SRS is operating in aggr mode (st_mode) or if there are
1016 * multiple Tx rings owned by Tx SRS, then each Tx ring (pseudo or
1017 * otherwise) will have a soft ring associated with it. These soft rings
1018 * are stored in srs_tx_soft_rings[] array.
1019 *
1020 * Additionally in the case of aggr, there is the st_soft_rings[] array
1021 * in the mac_srs_tx_t structure. This array is used to store the same
1022 * set of soft rings that are present in srs_tx_soft_rings[] array but
1023 * in a different manner. The soft ring associated with the pseudo Tx
1024 * ring is saved at mr_index (of the pseudo ring) in st_soft_rings[]
1025 * array. This helps in quickly getting the soft ring associated with the
1026 * Tx ring when aggr_find_tx_ring() returns the pseudo Tx ring that is to
1027 * be used for transmit.
1028 */
1029 mac_tx_mode_t mac_tx_mode_list[] = {
1030 {SRS_TX_DEFAULT, mac_tx_single_ring_mode},
1031 {SRS_TX_SERIALIZE, mac_tx_serializer_mode},
1032 {SRS_TX_FANOUT, mac_tx_fanout_mode},
1033 {SRS_TX_BW, mac_tx_bw_mode},
1034 {SRS_TX_BW_FANOUT, mac_tx_bw_mode},
1035 {SRS_TX_AGGR, mac_tx_aggr_mode},
1036 {SRS_TX_BW_AGGR, mac_tx_bw_mode}
1037 };
1038
1039 /*
1040 * Soft Ring Set (SRS) - The Run time code that deals with
1041 * dynamic polling from the hardware, bandwidth enforcement,
1042 * fanout etc.
1043 *
1044 * We try to use H/W classification on NIC and assign traffic for
1045 * a MAC address to a particular Rx ring or ring group. There is a
1046 * 1-1 mapping between a SRS and a Rx ring. The SRS dynamically
1047 * switches the underlying Rx ring between interrupt and
1048 * polling mode and enforces any specified B/W control.
1049 *
1050 * There is always a SRS created and tied to each H/W and S/W rule.
1051 * Whenever we create a H/W rule, we always add the the same rule to
1052 * S/W classifier and tie a SRS to it.
1053 *
1054 * In case a B/W control is specified, it is broken into bytes
1055 * per ticks and as soon as the quota for a tick is exhausted,
1056 * the underlying Rx ring is forced into poll mode for remainder of
1057 * the tick. The SRS poll thread only polls for bytes that are
1058 * allowed to come in the SRS. We typically let 4x the configured
1059 * B/W worth of packets to come in the SRS (to prevent unnecessary
1060 * drops due to bursts) but only process the specified amount.
1061 *
1062 * A MAC client (e.g. a VNIC or aggr) can have 1 or more
1063 * Rx rings (and corresponding SRSs) assigned to it. The SRS
1064 * in turn can have softrings to do protocol level fanout or
1065 * softrings to do S/W based fanout or both. In case the NIC
1066 * has no Rx rings, we do S/W classification to respective SRS.
1067 * The S/W classification rule is always setup and ready. This
1068 * allows the MAC layer to reassign Rx rings whenever needed
1069 * but packets still continue to flow via the default path and
1070 * getting S/W classified to correct SRS.
1071 *
1072 * The SRS's are used on both Tx and Rx side. They use the same
1073 * data structure but the processing routines have slightly different
1074 * semantics due to the fact that Rx side needs to do dynamic
1075 * polling etc.
1076 *
1077 * Dynamic Polling Notes
1078 * =====================
1079 *
1080 * Each Soft ring set is capable of switching its Rx ring between
1081 * interrupt and poll mode and actively 'polls' for packets in
1082 * poll mode. If the SRS is implementing a B/W limit, it makes
1083 * sure that only Max allowed packets are pulled in poll mode
1084 * and goes to poll mode as soon as B/W limit is exceeded. As
1085 * such, there are no overheads to implement B/W limits.
1086 *
1087 * In poll mode, its better to keep the pipeline going where the
1088 * SRS worker thread keeps processing packets and poll thread
1089 * keeps bringing more packets (specially if they get to run
1090 * on different CPUs). This also prevents the overheads associated
1091 * by excessive signalling (on NUMA machines, this can be
1092 * pretty devastating). The exception is latency optimized case
1093 * where worker thread does no work and interrupt and poll thread
1094 * are allowed to do their own drain.
1095 *
1096 * We use the following policy to control Dynamic Polling:
1097 * 1) We switch to poll mode anytime the processing
1098 * thread causes a backlog to build up in SRS and
1099 * its associated Soft Rings (sr_poll_pkt_cnt > 0).
1100 * 2) As long as the backlog stays under the low water
1101 * mark (sr_lowat), we poll the H/W for more packets.
1102 * 3) If the backlog (sr_poll_pkt_cnt) exceeds low
1103 * water mark, we stay in poll mode but don't poll
1104 * the H/W for more packets.
1105 * 4) Anytime in polling mode, if we poll the H/W for
1106 * packets and find nothing plus we have an existing
1107 * backlog (sr_poll_pkt_cnt > 0), we stay in polling
1108 * mode but don't poll the H/W for packets anymore
1109 * (let the polling thread go to sleep).
1110 * 5) Once the backlog is relived (packets are processed)
1111 * we reenable polling (by signalling the poll thread)
1112 * only when the backlog dips below sr_poll_thres.
1113 * 6) sr_hiwat is used exclusively when we are not
1114 * polling capable and is used to decide when to
1115 * drop packets so the SRS queue length doesn't grow
1116 * infinitely.
1117 *
1118 * NOTE: Also see the block level comment on top of mac_soft_ring.c
1119 */
1120
1121 /*
1122 * mac_latency_optimize
1123 *
1124 * Controls whether the poll thread can process the packets inline
1125 * or let the SRS worker thread do the processing. This applies if
1126 * the SRS was not being processed. For latency sensitive traffic,
1127 * this needs to be true to allow inline processing. For throughput
1128 * under load, this should be false.
1129 *
1130 * This (and other similar) tunable should be rolled into a link
1131 * or flow specific workload hint that can be set using dladm
1132 * linkprop (instead of multiple such tunables).
1133 */
1134 boolean_t mac_latency_optimize = B_TRUE;
1135
1136 /*
1137 * MAC_RX_SRS_ENQUEUE_CHAIN and MAC_TX_SRS_ENQUEUE_CHAIN
1138 *
1139 * queue a mp or chain in soft ring set and increment the
1140 * local count (srs_count) for the SRS and the shared counter
1141 * (srs_poll_pkt_cnt - shared between SRS and its soft rings
1142 * to track the total unprocessed packets for polling to work
1143 * correctly).
1144 *
1145 * The size (total bytes queued) counters are incremented only
1146 * if we are doing B/W control.
1147 */
1148 #define MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1149 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1150 if ((mac_srs)->srs_last != NULL) \
1151 (mac_srs)->srs_last->b_next = (head); \
1152 else \
1153 (mac_srs)->srs_first = (head); \
1154 (mac_srs)->srs_last = (tail); \
1155 (mac_srs)->srs_count += count; \
1156 }
1157
1158 #define MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1159 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \
1160 \
1161 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \
1162 srs_rx->sr_poll_pkt_cnt += count; \
1163 ASSERT(srs_rx->sr_poll_pkt_cnt > 0); \
1164 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \
1165 (mac_srs)->srs_size += (sz); \
1166 mutex_enter(&(mac_srs)->srs_bw->mac_bw_lock); \
1167 (mac_srs)->srs_bw->mac_bw_sz += (sz); \
1168 mutex_exit(&(mac_srs)->srs_bw->mac_bw_lock); \
1169 } \
1170 }
1171
1172 #define MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1173 mac_srs->srs_state |= SRS_ENQUEUED; \
1174 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \
1175 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \
1176 (mac_srs)->srs_size += (sz); \
1177 (mac_srs)->srs_bw->mac_bw_sz += (sz); \
1178 } \
1179 }
1180
1181 /*
1182 * Turn polling on routines
1183 */
1184 #define MAC_SRS_POLLING_ON(mac_srs) { \
1185 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1186 if (((mac_srs)->srs_state & \
1187 (SRS_POLLING_CAPAB|SRS_POLLING)) == SRS_POLLING_CAPAB) { \
1188 (mac_srs)->srs_state |= SRS_POLLING; \
1189 (void) mac_hwring_disable_intr((mac_ring_handle_t) \
1190 (mac_srs)->srs_ring); \
1191 (mac_srs)->srs_rx.sr_poll_on++; \
1192 } \
1193 }
1194
1195 #define MAC_SRS_WORKER_POLLING_ON(mac_srs) { \
1196 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1197 if (((mac_srs)->srs_state & \
1198 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_POLLING)) == \
1199 (SRS_POLLING_CAPAB|SRS_WORKER)) { \
1200 (mac_srs)->srs_state |= SRS_POLLING; \
1201 (void) mac_hwring_disable_intr((mac_ring_handle_t) \
1202 (mac_srs)->srs_ring); \
1203 (mac_srs)->srs_rx.sr_worker_poll_on++; \
1204 } \
1205 }
1206
1207 /*
1208 * MAC_SRS_POLL_RING
1209 *
1210 * Signal the SRS poll thread to poll the underlying H/W ring
1211 * provided it wasn't already polling (SRS_GET_PKTS was set).
1212 *
1213 * Poll thread gets to run only from mac_rx_srs_drain() and only
1214 * if the drain was being done by the worker thread.
1215 */
1216 #define MAC_SRS_POLL_RING(mac_srs) { \
1217 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \
1218 \
1219 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1220 srs_rx->sr_poll_thr_sig++; \
1221 if (((mac_srs)->srs_state & \
1222 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_GET_PKTS)) == \
1223 (SRS_WORKER|SRS_POLLING_CAPAB)) { \
1224 (mac_srs)->srs_state |= SRS_GET_PKTS; \
1225 cv_signal(&(mac_srs)->srs_cv); \
1226 } else { \
1227 srs_rx->sr_poll_thr_busy++; \
1228 } \
1229 }
1230
1231 /*
1232 * MAC_SRS_CHECK_BW_CONTROL
1233 *
1234 * Check to see if next tick has started so we can reset the
1235 * SRS_BW_ENFORCED flag and allow more packets to come in the
1236 * system.
1237 */
1238 #define MAC_SRS_CHECK_BW_CONTROL(mac_srs) { \
1239 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1240 ASSERT(((mac_srs)->srs_type & SRST_TX) || \
1241 MUTEX_HELD(&(mac_srs)->srs_bw->mac_bw_lock)); \
1242 clock_t now = ddi_get_lbolt(); \
1243 if ((mac_srs)->srs_bw->mac_bw_curr_time != now) { \
1244 (mac_srs)->srs_bw->mac_bw_curr_time = now; \
1245 (mac_srs)->srs_bw->mac_bw_used = 0; \
1246 if ((mac_srs)->srs_bw->mac_bw_state & SRS_BW_ENFORCED) \
1247 (mac_srs)->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; \
1248 } \
1249 }
1250
1251 /*
1252 * MAC_SRS_WORKER_WAKEUP
1253 *
1254 * Wake up the SRS worker thread to process the queue as long as
1255 * no one else is processing the queue. If we are optimizing for
1256 * latency, we wake up the worker thread immediately or else we
1257 * wait mac_srs_worker_wakeup_ticks before worker thread gets
1258 * woken up.
1259 */
1260 int mac_srs_worker_wakeup_ticks = 0;
1261 #define MAC_SRS_WORKER_WAKEUP(mac_srs) { \
1262 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1263 if (!((mac_srs)->srs_state & SRS_PROC) && \
1264 (mac_srs)->srs_tid == NULL) { \
1265 if (((mac_srs)->srs_state & SRS_LATENCY_OPT) || \
1266 (mac_srs_worker_wakeup_ticks == 0)) \
1267 cv_signal(&(mac_srs)->srs_async); \
1268 else \
1269 (mac_srs)->srs_tid = \
1270 timeout(mac_srs_fire, (mac_srs), \
1271 mac_srs_worker_wakeup_ticks); \
1272 } \
1273 }
1274
1275 #define TX_BANDWIDTH_MODE(mac_srs) \
1276 ((mac_srs)->srs_tx.st_mode == SRS_TX_BW || \
1277 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_FANOUT || \
1278 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_AGGR)
1279
1280 #define TX_SRS_TO_SOFT_RING(mac_srs, head, hint) { \
1281 if (tx_mode == SRS_TX_BW_FANOUT) \
1282 (void) mac_tx_fanout_mode(mac_srs, head, hint, 0, NULL);\
1283 else \
1284 (void) mac_tx_aggr_mode(mac_srs, head, hint, 0, NULL); \
1285 }
1286
1287 /*
1288 * MAC_TX_SRS_BLOCK
1289 *
1290 * Always called from mac_tx_srs_drain() function. SRS_TX_BLOCKED
1291 * will be set only if srs_tx_woken_up is FALSE. If
1292 * srs_tx_woken_up is TRUE, it indicates that the wakeup arrived
1293 * before we grabbed srs_lock to set SRS_TX_BLOCKED. We need to
1294 * attempt to transmit again and not setting SRS_TX_BLOCKED does
1295 * that.
1296 */
1297 #define MAC_TX_SRS_BLOCK(srs, mp) { \
1298 ASSERT(MUTEX_HELD(&(srs)->srs_lock)); \
1299 if ((srs)->srs_tx.st_woken_up) { \
1300 (srs)->srs_tx.st_woken_up = B_FALSE; \
1301 } else { \
1302 ASSERT(!((srs)->srs_state & SRS_TX_BLOCKED)); \
1303 (srs)->srs_state |= SRS_TX_BLOCKED; \
1304 (srs)->srs_tx.st_stat.mts_blockcnt++; \
1305 } \
1306 }
1307
1308 /*
1309 * MAC_TX_SRS_TEST_HIWAT
1310 *
1311 * Called before queueing a packet onto Tx SRS to test and set
1312 * SRS_TX_HIWAT if srs_count exceeds srs_tx_hiwat.
1313 */
1314 #define MAC_TX_SRS_TEST_HIWAT(srs, mp, tail, cnt, sz, cookie) { \
1315 boolean_t enqueue = 1; \
1316 \
1317 if ((srs)->srs_count > (srs)->srs_tx.st_hiwat) { \
1318 /* \
1319 * flow-controlled. Store srs in cookie so that it \
1320 * can be returned as mac_tx_cookie_t to client \
1321 */ \
1322 (srs)->srs_state |= SRS_TX_HIWAT; \
1323 cookie = (mac_tx_cookie_t)srs; \
1324 (srs)->srs_tx.st_hiwat_cnt++; \
1325 if ((srs)->srs_count > (srs)->srs_tx.st_max_q_cnt) { \
1326 /* increment freed stats */ \
1327 (srs)->srs_tx.st_stat.mts_sdrops += cnt; \
1328 /* \
1329 * b_prev may be set to the fanout hint \
1330 * hence can't use freemsg directly \
1331 */ \
1332 mac_drop_chain(mp_chain, "SRS Tx max queue"); \
1333 DTRACE_PROBE1(tx_queued_hiwat, \
1334 mac_soft_ring_set_t *, srs); \
1335 enqueue = 0; \
1336 } \
1337 } \
1338 if (enqueue) \
1339 MAC_TX_SRS_ENQUEUE_CHAIN(srs, mp, tail, cnt, sz); \
1340 }
1341
1342 /* Some utility macros */
1343 #define MAC_SRS_BW_LOCK(srs) \
1344 if (!(srs->srs_type & SRST_TX)) \
1345 mutex_enter(&srs->srs_bw->mac_bw_lock);
1346
1347 #define MAC_SRS_BW_UNLOCK(srs) \
1348 if (!(srs->srs_type & SRST_TX)) \
1349 mutex_exit(&srs->srs_bw->mac_bw_lock);
1350
1351 #define MAC_TX_SRS_DROP_MESSAGE(srs, chain, cookie, s) { \
1352 mac_drop_chain((chain), (s)); \
1353 /* increment freed stats */ \
1354 (srs)->srs_tx.st_stat.mts_sdrops++; \
1355 (cookie) = (mac_tx_cookie_t)(srs); \
1356 }
1357
1358 #define MAC_TX_SET_NO_ENQUEUE(srs, mp_chain, ret_mp, cookie) { \
1359 mac_srs->srs_state |= SRS_TX_WAKEUP_CLIENT; \
1360 cookie = (mac_tx_cookie_t)srs; \
1361 *ret_mp = mp_chain; \
1362 }
1363
1364 /*
1365 * Threshold used in receive-side processing to determine if handling
1366 * can occur in situ (in the interrupt thread) or if it should be left to a
1367 * worker thread. Note that the constant used to make this determination is
1368 * not entirely made-up, and is a result of some emprical validation. That
1369 * said, the constant is left as a global variable to allow it to be
1370 * dynamically tuned in the field if and as needed.
1371 */
1372 uintptr_t mac_rx_srs_stack_needed = 14336;
1373 uint_t mac_rx_srs_stack_toodeep;
1374
1375 #ifndef STACK_GROWTH_DOWN
1376 #error Downward stack growth assumed.
1377 #endif
1378
1379 /*
1380 * Drop the rx packet and advance to the next one in the chain.
1381 */
1382 static void
mac_rx_drop_pkt(mac_soft_ring_set_t * srs,mblk_t * mp)1383 mac_rx_drop_pkt(mac_soft_ring_set_t *srs, mblk_t *mp)
1384 {
1385 mac_srs_rx_t *srs_rx = &srs->srs_rx;
1386
1387 ASSERT(mp->b_next == NULL);
1388 mutex_enter(&srs->srs_lock);
1389 MAC_UPDATE_SRS_COUNT_LOCKED(srs, 1);
1390 MAC_UPDATE_SRS_SIZE_LOCKED(srs, msgdsize(mp));
1391 mutex_exit(&srs->srs_lock);
1392
1393 srs_rx->sr_stat.mrs_sdrops++;
1394 freemsg(mp);
1395 }
1396
1397 /* DATAPATH RUNTIME ROUTINES */
1398
1399 /*
1400 * mac_srs_fire
1401 *
1402 * Timer callback routine for waking up the SRS worker thread.
1403 */
1404 static void
mac_srs_fire(void * arg)1405 mac_srs_fire(void *arg)
1406 {
1407 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)arg;
1408
1409 mutex_enter(&mac_srs->srs_lock);
1410 if (mac_srs->srs_tid == NULL) {
1411 mutex_exit(&mac_srs->srs_lock);
1412 return;
1413 }
1414
1415 mac_srs->srs_tid = NULL;
1416 if (!(mac_srs->srs_state & SRS_PROC))
1417 cv_signal(&mac_srs->srs_async);
1418
1419 mutex_exit(&mac_srs->srs_lock);
1420 }
1421
1422 /*
1423 * 'hint' is fanout_hint (type of uint64_t) which is given by the TCP/IP stack,
1424 * and it is used on the TX path.
1425 */
1426 #define HASH_HINT(hint) \
1427 ((hint) ^ ((hint) >> 24) ^ ((hint) >> 16) ^ ((hint) >> 8))
1428
1429
1430 /*
1431 * hash based on the src address, dst address and the port information.
1432 */
1433 #define HASH_ADDR(src, dst, ports) \
1434 (ntohl((src) + (dst)) ^ ((ports) >> 24) ^ ((ports) >> 16) ^ \
1435 ((ports) >> 8) ^ (ports))
1436
1437 /*
1438 * Uniform distribution hash for IPv6 4-tuple.
1439 */
1440 #define HASH_ADDR6(src, dst, ports) \
1441 ((src.s6_addr32[0] ^ src.s6_addr32[1] ^ \
1442 src.s6_addr32[2] ^ src.s6_addr32[3]) ^ \
1443 (dst.s6_addr32[0] ^ dst.s6_addr32[1] ^ \
1444 dst.s6_addr32[2] ^ dst.s6_addr32[3]) ^ \
1445 ((ports) >> 24) ^ ((ports) >> 16) ^ \
1446 ((ports) >> 8) ^ (ports))
1447
1448 #define COMPUTE_INDEX(key, sz) (key % sz)
1449
1450 #define FANOUT_ENQUEUE_MP(head, tail, cnt, bw_ctl, sz, sz0, mp) { \
1451 if ((tail) != NULL) { \
1452 ASSERT((tail)->b_next == NULL); \
1453 (tail)->b_next = (mp); \
1454 } else { \
1455 ASSERT((head) == NULL); \
1456 (head) = (mp); \
1457 } \
1458 (tail) = (mp); \
1459 (cnt)++; \
1460 if ((bw_ctl)) \
1461 (sz) += (sz0); \
1462 }
1463
1464 #define MAC_FANOUT_DEFAULT 0
1465 #define MAC_FANOUT_RND_ROBIN 1
1466 int mac_fanout_type = MAC_FANOUT_DEFAULT;
1467
1468 #define MAX_SR_TYPES 5
1469 /* fanout types for port based hashing */
1470 typedef enum pkt_type {
1471 V4_TCP = 0,
1472 V4_UDP,
1473 V6_TCP,
1474 V6_UDP,
1475 OTH,
1476 UNDEF
1477 } pkt_type_t;
1478
1479 /*
1480 * Pair of local and remote ports in the transport header
1481 */
1482 #define PORTS_SIZE 4
1483
1484 /*
1485 * This routine delivers packets destined for an SRS into one of the
1486 * protocol soft rings.
1487 *
1488 * Given a chain of packets we need to split it up into multiple sub
1489 * chains: TCP, UDP or OTH soft ring. Instead of entering the soft
1490 * ring one packet at a time, we want to enter it in the form of a
1491 * chain otherwise we get this start/stop behaviour where the worker
1492 * thread goes to sleep and then next packet comes in forcing it to
1493 * wake up.
1494 */
1495 static void
mac_rx_srs_proto_fanout(mac_soft_ring_set_t * mac_srs,mblk_t * head)1496 mac_rx_srs_proto_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head)
1497 {
1498 mblk_t *headmp[MAX_SR_TYPES] = { 0 };
1499 mblk_t *tailmp[MAX_SR_TYPES] = { 0 };
1500 int cnt[MAX_SR_TYPES] = { 0 };
1501 size_t sz[MAX_SR_TYPES] = { 0 };
1502 mac_client_impl_t *mcip = mac_srs->srs_mcip;
1503
1504 const boolean_t is_ether =
1505 (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER);
1506 const boolean_t bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0);
1507
1508 /*
1509 * If we don't have a Rx ring, S/W classification would have done
1510 * its job and its a packet meant for us. If we were polling on
1511 * the default ring (i.e. there was a ring assigned to this SRS),
1512 * then we need to make sure that the mac address really belongs
1513 * to us.
1514 */
1515 const boolean_t hw_classified = mac_srs->srs_ring != NULL &&
1516 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER;
1517
1518 /*
1519 * Some clients, such as non-ethernet, need DLS processing in
1520 * the Rx path. Such clients clear the SRST_DLS_BYPASS flag.
1521 * DLS bypass may also be disabled via the
1522 * MCIS_RX_BYPASS_DISABLE flag.
1523 */
1524 const boolean_t dls_bypass =
1525 ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) &&
1526 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0);
1527
1528 /*
1529 * We have a chain from SRS that we need to split across the
1530 * soft rings. The squeues for the TCP and IPv4 SAPs use their
1531 * own soft rings to allow polling from the squeue. The rest of
1532 * the packets are delivered on the OTH soft ring which cannot
1533 * be polled.
1534 */
1535 while (head != NULL) {
1536 mac_ether_offload_info_t meoi = { 0 };
1537 uint8_t ether_addr[ETHERADDRL];
1538 const uint8_t *dstaddr = ether_addr;
1539 mac_header_info_t non_ether_mhi;
1540 boolean_t is_unicast = B_FALSE;
1541
1542 mblk_t *mp = head;
1543 head = head->b_next;
1544 mp->b_next = NULL;
1545 const size_t sz1 =
1546 (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp);
1547
1548 if (is_ether) {
1549 uint32_t vlan_tci;
1550
1551 mac_ether_offload_info(mp, &meoi);
1552 if ((meoi.meoi_flags & MEOI_L2INFO_SET) == 0 ||
1553 !mac_ether_l2_info(mp, ether_addr, &vlan_tci)) {
1554 mac_rx_drop_pkt(mac_srs, mp);
1555 continue;
1556 }
1557
1558 /*
1559 * Check if the VID of the packet, if any, belongs to
1560 * this client. Technically, if this packet came up via
1561 * a HW classified ring then we don't need to perform
1562 * this check. Perhaps a future optimization.
1563 */
1564 if ((meoi.meoi_flags & MEOI_VLAN_TAGGED) != 0) {
1565 ASSERT3U(meoi.meoi_l2hlen, ==,
1566 sizeof (struct ether_vlan_header));
1567 ASSERT3U(vlan_tci, <=, UINT16_MAX);
1568
1569 if (!mac_client_check_flow_vid(mcip,
1570 VLAN_ID(vlan_tci))) {
1571 mac_rx_drop_pkt(mac_srs, mp);
1572 continue;
1573 }
1574 }
1575
1576 is_unicast = ((ether_addr[0] & 0x01) == 0);
1577 } else {
1578 if (mac_header_info((mac_handle_t)mcip->mci_mip,
1579 mp, &non_ether_mhi) != 0) {
1580 mac_rx_drop_pkt(mac_srs, mp);
1581 continue;
1582 }
1583
1584 meoi.meoi_l2hlen = non_ether_mhi.mhi_hdrsize;
1585 meoi.meoi_l3proto = non_ether_mhi.mhi_bindsap;
1586 meoi.meoi_flags = MEOI_L2INFO_SET;
1587 (void) mac_partial_offload_info(mp, 0, &meoi);
1588
1589 is_unicast =
1590 (non_ether_mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST);
1591 dstaddr = non_ether_mhi.mhi_daddr;
1592 }
1593
1594 if (!dls_bypass) {
1595 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
1596 mac_ether_offload_info_t *, &meoi,
1597 mac_soft_ring_set_t *, mac_srs, pkt_type_t, OTH);
1598 FANOUT_ENQUEUE_MP(headmp[OTH], tailmp[OTH],
1599 cnt[OTH], bw_ctl, sz[OTH], sz1, mp);
1600 continue;
1601 }
1602
1603 ASSERT((meoi.meoi_flags & MEOI_L2INFO_SET) != 0);
1604
1605 boolean_t is_fastpath = B_FALSE;
1606
1607 if (meoi.meoi_l3proto == ETHERTYPE_IP ||
1608 meoi.meoi_l3proto == ETHERTYPE_IPV6) {
1609 /*
1610 * If we are H/W classified, but we have promisc
1611 * on, then we need to check for the unicast address.
1612 */
1613 if (hw_classified && mcip->mci_promisc_list != NULL) {
1614 mac_address_t *map;
1615
1616 rw_enter(&mcip->mci_rw_lock, RW_READER);
1617 map = mcip->mci_unicast;
1618 if (bcmp(dstaddr, map->ma_addr,
1619 map->ma_len) == 0)
1620 is_fastpath = B_TRUE;
1621 rw_exit(&mcip->mci_rw_lock);
1622 } else if (is_unicast) {
1623 is_fastpath = B_TRUE;
1624 }
1625 }
1626
1627 /*
1628 * This needs to become a contract with the driver for
1629 * the fast path.
1630 *
1631 * In the normal case the packet will have at least the L2
1632 * header and the IP + Transport header in the same mblk.
1633 * This is usually the case when the NIC driver sends up
1634 * the packet. This is also true when the stack generates
1635 * a packet that is looped back and when the stack uses the
1636 * fastpath mechanism. The normal case is optimized for
1637 * performance and may bypass DLS. All other cases go through
1638 * the 'OTH' type path without DLS bypass.
1639 */
1640 if (is_fastpath) {
1641 if ((meoi.meoi_flags & MEOI_L3INFO_SET) == 0 ||
1642 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0) {
1643 is_fastpath = B_FALSE;
1644 }
1645 if (DB_TYPE(mp) != M_DATA || DB_REF(mp) != 1) {
1646 is_fastpath = B_FALSE;
1647 }
1648
1649 const size_t total_hdr_len = meoi.meoi_l2hlen
1650 + meoi.meoi_l3hlen + meoi.meoi_l4hlen;
1651
1652 if (!OK_32PTR(mp->b_rptr + meoi.meoi_l2hlen) ||
1653 total_hdr_len > MBLKL(mp)) {
1654 is_fastpath = B_FALSE;
1655 }
1656 }
1657
1658 if (!is_fastpath) {
1659 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
1660 mac_ether_offload_info_t *, &meoi,
1661 mac_soft_ring_set_t *, mac_srs, pkt_type_t, OTH);
1662 FANOUT_ENQUEUE_MP(headmp[OTH], tailmp[OTH],
1663 cnt[OTH], bw_ctl, sz[OTH], sz1, mp);
1664 continue;
1665 }
1666
1667 /*
1668 * Determine the type from the IP protocol value. If classified
1669 * as TCP or UDP, then update the read pointer to the beginning
1670 * of the IP header. Otherwise leave the message as is for
1671 * further processing by DLS.
1672 */
1673 pkt_type_t type = OTH;
1674 switch (meoi.meoi_l4proto) {
1675 case IPPROTO_TCP:
1676 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ?
1677 V6_TCP : V4_TCP;
1678 mp->b_rptr += meoi.meoi_l2hlen;
1679 break;
1680 case IPPROTO_UDP:
1681 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ?
1682 V6_UDP : V4_UDP;
1683 mp->b_rptr += meoi.meoi_l2hlen;
1684 break;
1685 default:
1686 break;
1687 }
1688
1689 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
1690 mac_ether_offload_info_t *, &meoi, mac_soft_ring_set_t *,
1691 mac_srs, pkt_type_t, type);
1692 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], cnt[type],
1693 bw_ctl, sz[type], sz1, mp);
1694 }
1695
1696 for (pkt_type_t type = V4_TCP; type < UNDEF; type++) {
1697 if (headmp[type] != NULL) {
1698 mac_soft_ring_t *softring;
1699
1700 ASSERT(tailmp[type]->b_next == NULL);
1701 switch (type) {
1702 case V4_TCP:
1703 softring = mac_srs->srs_tcp_soft_rings[0];
1704 break;
1705 case V6_TCP:
1706 softring = mac_srs->srs_tcp6_soft_rings[0];
1707 break;
1708 case V4_UDP:
1709 softring = mac_srs->srs_udp_soft_rings[0];
1710 break;
1711 case V6_UDP:
1712 softring = mac_srs->srs_udp6_soft_rings[0];
1713 break;
1714 case OTH:
1715 softring = mac_srs->srs_oth_soft_rings[0];
1716 }
1717 mac_rx_soft_ring_process(mcip, softring,
1718 headmp[type], tailmp[type], cnt[type], sz[type]);
1719 }
1720 }
1721 }
1722
1723 int fanout_unaligned = 0;
1724
1725 /*
1726 * The fanout routine for any clients with DLS bypass disabled or for
1727 * traffic classified as "other". Returns -1 on an error (drop the
1728 * packet due to a malformed packet), 0 on success, with values
1729 * written in *indx and *type.
1730 */
1731 static int
mac_rx_srs_long_fanout(mac_soft_ring_set_t * mac_srs,mblk_t * mp,uint32_t sap,size_t hdrsize,pkt_type_t * type,uint_t * indx)1732 mac_rx_srs_long_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *mp,
1733 uint32_t sap, size_t hdrsize, pkt_type_t *type, uint_t *indx)
1734 {
1735 ip6_t *ip6h;
1736 ipha_t *ipha;
1737 uint8_t *whereptr;
1738 uint_t hash;
1739 uint16_t remlen;
1740 uint8_t nexthdr;
1741 uint16_t hdr_len;
1742 uint32_t src_val, dst_val;
1743 boolean_t modifiable = B_TRUE;
1744 boolean_t v6;
1745
1746 ASSERT(MBLKL(mp) >= hdrsize);
1747
1748 if (sap == ETHERTYPE_IPV6) {
1749 v6 = B_TRUE;
1750 hdr_len = IPV6_HDR_LEN;
1751 } else if (sap == ETHERTYPE_IP) {
1752 v6 = B_FALSE;
1753 hdr_len = IP_SIMPLE_HDR_LENGTH;
1754 } else {
1755 *indx = 0;
1756 *type = OTH;
1757 return (0);
1758 }
1759
1760 ip6h = (ip6_t *)(mp->b_rptr + hdrsize);
1761 ipha = (ipha_t *)ip6h;
1762
1763 if ((uint8_t *)ip6h == mp->b_wptr) {
1764 /*
1765 * The first mblk_t only includes the mac header.
1766 * Note that it is safe to change the mp pointer here,
1767 * as the subsequent operation does not assume mp
1768 * points to the start of the mac header.
1769 */
1770 mp = mp->b_cont;
1771
1772 /*
1773 * Make sure the IP header points to an entire one.
1774 */
1775 if (mp == NULL)
1776 return (-1);
1777
1778 if (MBLKL(mp) < hdr_len) {
1779 modifiable = (DB_REF(mp) == 1);
1780
1781 if (modifiable && !pullupmsg(mp, hdr_len))
1782 return (-1);
1783 }
1784
1785 ip6h = (ip6_t *)mp->b_rptr;
1786 ipha = (ipha_t *)ip6h;
1787 }
1788
1789 if (!modifiable || !(OK_32PTR((char *)ip6h)) ||
1790 ((uint8_t *)ip6h + hdr_len > mp->b_wptr)) {
1791 /*
1792 * If either the IP header is not aligned, or it does not hold
1793 * the complete simple structure (a pullupmsg() is not an
1794 * option since it would result in an unaligned IP header),
1795 * fanout to the default ring.
1796 *
1797 * Note that this may cause packet reordering.
1798 */
1799 *indx = 0;
1800 *type = OTH;
1801 fanout_unaligned++;
1802 return (0);
1803 }
1804
1805 /*
1806 * Extract next-header, full header length, and source-hash value
1807 * using v4/v6 specific fields.
1808 */
1809 if (v6) {
1810 remlen = ntohs(ip6h->ip6_plen);
1811 nexthdr = ip6h->ip6_nxt;
1812 src_val = V4_PART_OF_V6(ip6h->ip6_src);
1813 dst_val = V4_PART_OF_V6(ip6h->ip6_dst);
1814 /*
1815 * Do src based fanout if below tunable is set to B_TRUE or
1816 * when mac_ip_hdr_length_v6() fails because of malformed
1817 * packets or because mblks need to be concatenated using
1818 * pullupmsg().
1819 *
1820 * Perform a version check to prevent parsing weirdness...
1821 */
1822 if (IPH_HDR_VERSION(ip6h) != IPV6_VERSION ||
1823 !mac_ip_hdr_length_v6(ip6h, mp->b_wptr, &hdr_len, &nexthdr,
1824 NULL)) {
1825 goto src_dst_based_fanout;
1826 }
1827 } else {
1828 hdr_len = IPH_HDR_LENGTH(ipha);
1829 remlen = ntohs(ipha->ipha_length) - hdr_len;
1830 nexthdr = ipha->ipha_protocol;
1831 src_val = (uint32_t)ipha->ipha_src;
1832 dst_val = (uint32_t)ipha->ipha_dst;
1833 /*
1834 * Catch IPv4 fragment case here. IPv6 has nexthdr == FRAG
1835 * for its equivalent case.
1836 */
1837 if ((ntohs(ipha->ipha_fragment_offset_and_flags) &
1838 (IPH_MF | IPH_OFFSET)) != 0) {
1839 goto src_dst_based_fanout;
1840 }
1841 }
1842 if (remlen < MIN_EHDR_LEN)
1843 return (-1);
1844 whereptr = (uint8_t *)ip6h + hdr_len;
1845
1846 /* If the transport is one of below, we do port/SPI based fanout */
1847 switch (nexthdr) {
1848 case IPPROTO_TCP:
1849 case IPPROTO_UDP:
1850 case IPPROTO_SCTP:
1851 case IPPROTO_ESP:
1852 /*
1853 * If the ports or SPI in the transport header is not part of
1854 * the mblk, do src_based_fanout, instead of calling
1855 * pullupmsg().
1856 */
1857 if (mp->b_cont == NULL || whereptr + PORTS_SIZE <= mp->b_wptr)
1858 break; /* out of switch... */
1859 /* FALLTHRU */
1860 default:
1861 goto src_dst_based_fanout;
1862 }
1863
1864 switch (nexthdr) {
1865 case IPPROTO_TCP:
1866 hash = HASH_ADDR(src_val, dst_val, *(uint32_t *)whereptr);
1867 *indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count);
1868 *type = OTH;
1869 break;
1870 case IPPROTO_UDP:
1871 case IPPROTO_SCTP:
1872 case IPPROTO_ESP:
1873 if (mac_fanout_type == MAC_FANOUT_DEFAULT) {
1874 hash = HASH_ADDR(src_val, dst_val,
1875 *(uint32_t *)whereptr);
1876 *indx = COMPUTE_INDEX(hash,
1877 mac_srs->srs_udp_ring_count);
1878 } else {
1879 *indx = mac_srs->srs_ind % mac_srs->srs_udp_ring_count;
1880 mac_srs->srs_ind++;
1881 }
1882 *type = OTH;
1883 break;
1884 }
1885 return (0);
1886
1887 src_dst_based_fanout:
1888 hash = HASH_ADDR(src_val, dst_val, (uint32_t)0);
1889 *indx = COMPUTE_INDEX(hash, mac_srs->srs_oth_ring_count);
1890 *type = OTH;
1891 return (0);
1892 }
1893
1894 /*
1895 * This routine delivers packets destined for an SRS into a soft ring member
1896 * of the set.
1897 *
1898 * Given a chain of packets we need to split it up into multiple sub
1899 * chains: TCP, UDP or OTH soft ring. Instead of entering the soft
1900 * ring one packet at a time, we want to enter it in the form of a
1901 * chain otherwise we get this start/stop behaviour where the worker
1902 * thread goes to sleep and then next packet comes in forcing it to
1903 * wake up.
1904 *
1905 * Note:
1906 * Since we know what is the maximum fanout possible, we create a 2D array
1907 * of 'softring types * MAX_SR_FANOUT' for the head, tail, cnt and sz
1908 * variables so that we can enter the softrings with chain. We need the
1909 * MAX_SR_FANOUT so we can allocate the arrays on the stack (a kmem_alloc
1910 * for each packet would be expensive). If we ever want to have the
1911 * ability to have unlimited fanout, we should probably declare a head,
1912 * tail, cnt, sz with each soft ring (a data struct which contains a softring
1913 * along with these members) and create an array of this uber struct so we
1914 * don't have to do kmem_alloc.
1915 */
1916
1917 static void
mac_rx_srs_fanout(mac_soft_ring_set_t * mac_srs,mblk_t * head)1918 mac_rx_srs_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head)
1919 {
1920 mblk_t *headmp[MAX_SR_TYPES][MAX_SR_FANOUT];
1921 mblk_t *tailmp[MAX_SR_TYPES][MAX_SR_FANOUT];
1922 int cnt[MAX_SR_TYPES][MAX_SR_FANOUT];
1923 size_t sz[MAX_SR_TYPES][MAX_SR_FANOUT];
1924 mac_client_impl_t *mcip = mac_srs->srs_mcip;
1925
1926 const boolean_t is_ether =
1927 (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER);
1928 const boolean_t bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0);
1929
1930 /*
1931 * If we don't have a Rx ring, S/W classification would have done
1932 * its job and its a packet meant for us. If we were polling on
1933 * the default ring (i.e. there was a ring assigned to this SRS),
1934 * then we need to make sure that the mac address really belongs
1935 * to us.
1936 */
1937 const boolean_t hw_classified = mac_srs->srs_ring != NULL &&
1938 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER;
1939
1940 /*
1941 * Some clients, such as non Ethernet, need DLS processing in
1942 * the Rx path. Such clients clear the SRST_DLS_BYPASS flag.
1943 * DLS bypass may also be disabled via the
1944 * MCIS_RX_BYPASS_DISABLE flag, but this is only consumed by
1945 * sun4v vsw currently.
1946 */
1947 const boolean_t dls_bypass =
1948 ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) &&
1949 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0);
1950
1951 /*
1952 * Since the softrings are never destroyed and we always
1953 * create equal number of softrings for TCP, UDP and rest,
1954 * its OK to check one of them for count and use it without
1955 * any lock. In future, if soft rings get destroyed because
1956 * of reduction in fanout, we will need to ensure that happens
1957 * behind the SRS_PROC.
1958 */
1959 const int fanout_cnt = mac_srs->srs_tcp_ring_count;
1960
1961 bzero(headmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *));
1962 bzero(tailmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *));
1963 bzero(cnt, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (int));
1964 bzero(sz, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (size_t));
1965
1966 /*
1967 * We got a chain from SRS that we need to send to the soft rings.
1968 * Since squeues for TCP & IPv4 SAP poll their soft rings (for
1969 * performance reasons), we need to separate out v4_tcp, v4_udp
1970 * and the rest goes in other.
1971 */
1972 while (head != NULL) {
1973 mac_ether_offload_info_t meoi = { 0 };
1974 uint8_t ether_addr[ETHERADDRL];
1975 const uint8_t *dstaddr = ether_addr;
1976 mac_header_info_t non_ether_mhi;
1977 pkt_type_t type;
1978 uint_t indx;
1979 boolean_t is_unicast = B_FALSE;
1980
1981 mblk_t *mp = head;
1982 head = head->b_next;
1983 mp->b_next = NULL;
1984 const size_t sz1 =
1985 (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp);
1986
1987 if (is_ether) {
1988 uint32_t vlan_tci;
1989
1990 /*
1991 * At this point we can be sure the packet at least
1992 * has an ether header.
1993 */
1994 mac_ether_offload_info(mp, &meoi);
1995 if ((meoi.meoi_flags & MEOI_L2INFO_SET) == 0 ||
1996 !mac_ether_l2_info(mp, ether_addr, &vlan_tci)) {
1997 mac_rx_drop_pkt(mac_srs, mp);
1998 continue;
1999 }
2000
2001 /*
2002 * Check if the VID of the packet, if any, belongs to
2003 * this client. Technically, if this packet came up via
2004 * a HW classified ring then we don't need to perform
2005 * this check. Perhaps a future optimization.
2006 */
2007 if ((meoi.meoi_flags & MEOI_VLAN_TAGGED) != 0) {
2008 ASSERT3U(meoi.meoi_l2hlen, ==,
2009 sizeof (struct ether_vlan_header));
2010 ASSERT3U(vlan_tci, <=, UINT16_MAX);
2011
2012 if (!mac_client_check_flow_vid(mcip,
2013 VLAN_ID(vlan_tci))) {
2014 mac_rx_drop_pkt(mac_srs, mp);
2015 continue;
2016 }
2017 }
2018
2019 is_unicast = (ether_addr[0] & 0x01) == 0;
2020 } else {
2021 if (mac_header_info((mac_handle_t)mcip->mci_mip,
2022 mp, &non_ether_mhi) != 0) {
2023 mac_rx_drop_pkt(mac_srs, mp);
2024 continue;
2025 }
2026
2027 meoi.meoi_l2hlen = non_ether_mhi.mhi_hdrsize;
2028 meoi.meoi_l3proto = non_ether_mhi.mhi_bindsap;
2029 meoi.meoi_flags = MEOI_L2INFO_SET;
2030 (void) mac_partial_offload_info(mp, 0, &meoi);
2031
2032 is_unicast =
2033 (non_ether_mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST);
2034 dstaddr = non_ether_mhi.mhi_daddr;
2035 }
2036
2037 if (!dls_bypass) {
2038 if (mac_rx_srs_long_fanout(mac_srs, mp,
2039 meoi.meoi_l3proto, meoi.meoi_l2hlen,
2040 &type, &indx) == -1) {
2041 mac_rx_drop_pkt(mac_srs, mp);
2042 continue;
2043 }
2044
2045 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
2046 mac_ether_offload_info_t *, &meoi,
2047 mac_soft_ring_set_t *, mac_srs, pkt_type_t, type);
2048 FANOUT_ENQUEUE_MP(headmp[type][indx],
2049 tailmp[type][indx],
2050 cnt[type][indx], bw_ctl,
2051 sz[type][indx], sz1, mp);
2052 continue;
2053 }
2054
2055 /*
2056 * While MEOI is unable to parse ESP headers, for the purposes
2057 * of classification here, we treat such packets like UDP, so we
2058 * can grant it a reprieve here. This is acceptable since we
2059 * will not go rooting around in the ESP headers.
2060 */
2061 if ((meoi.meoi_flags & MEOI_L3INFO_SET) != 0 &&
2062 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0 &&
2063 meoi.meoi_l4proto == IPPROTO_ESP) {
2064 /* ESP header should consist of at least 8 octets */
2065 meoi.meoi_l4hlen = 8;
2066 meoi.meoi_flags |= MEOI_L4INFO_SET;
2067 }
2068
2069 /*
2070 * If we are using the default Rx ring where H/W or S/W
2071 * classification has not happened, we need to verify if
2072 * this unicast packet really belongs to us.
2073 */
2074 boolean_t is_fastpath = B_FALSE;
2075 if (meoi.meoi_l3proto == ETHERTYPE_IP ||
2076 meoi.meoi_l3proto == ETHERTYPE_IPV6) {
2077 /*
2078 * If we are H/W classified, but we have promisc
2079 * on, then we need to check for the unicast address.
2080 */
2081 if (hw_classified && mcip->mci_promisc_list != NULL) {
2082 mac_address_t *map;
2083
2084 rw_enter(&mcip->mci_rw_lock, RW_READER);
2085 map = mcip->mci_unicast;
2086 if (bcmp(dstaddr, map->ma_addr,
2087 map->ma_len) == 0)
2088 is_fastpath = B_TRUE;
2089 rw_exit(&mcip->mci_rw_lock);
2090 } else if (is_unicast) {
2091 is_fastpath = B_TRUE;
2092 }
2093 }
2094
2095 /*
2096 * Verify that the requirements for taking the fast path are all
2097 * still met. This needs to become a contract with the driver.
2098 */
2099 if (is_fastpath) {
2100 if ((meoi.meoi_flags & MEOI_L3INFO_SET) == 0 ||
2101 (meoi.meoi_flags & MEOI_L4INFO_SET) == 0) {
2102 is_fastpath = B_FALSE;
2103 }
2104 if (DB_TYPE(mp) != M_DATA || DB_REF(mp) != 1) {
2105 is_fastpath = B_FALSE;
2106 }
2107
2108 const size_t total_hdr_len = meoi.meoi_l2hlen
2109 + meoi.meoi_l3hlen + meoi.meoi_l4hlen;
2110
2111 if (!OK_32PTR(mp->b_rptr + meoi.meoi_l2hlen) ||
2112 total_hdr_len > MBLKL(mp)) {
2113 is_fastpath = B_FALSE;
2114 }
2115
2116 if ((meoi.meoi_flags &
2117 (MEOI_L3_FRAG_MORE | MEOI_L3_FRAG_OFFSET)) != 0) {
2118 is_fastpath = B_FALSE;
2119 }
2120 }
2121 switch (meoi.meoi_l4proto) {
2122 case IPPROTO_TCP:
2123 case IPPROTO_UDP:
2124 case IPPROTO_SCTP:
2125 case IPPROTO_ESP:
2126 if (is_fastpath) {
2127 /*
2128 * Since the above checks ensure that the first
2129 * mblk covers the L2-L4 headers, we can be
2130 * confident that the "ports" portion of the
2131 * hashing payload is covered too.
2132 */
2133 ASSERT3U(meoi.meoi_l4hlen, >=, PORTS_SIZE);
2134 }
2135 break;
2136 default:
2137 break;
2138 }
2139
2140 if (!is_fastpath) {
2141 if (mac_rx_srs_long_fanout(mac_srs, mp,
2142 meoi.meoi_l3proto, meoi.meoi_l2hlen,
2143 &type, &indx) == -1) {
2144 mac_rx_drop_pkt(mac_srs, mp);
2145 continue;
2146 }
2147
2148 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
2149 mac_ether_offload_info_t *, &meoi,
2150 mac_soft_ring_set_t *, mac_srs, pkt_type_t, type);
2151 FANOUT_ENQUEUE_MP(headmp[type][indx],
2152 tailmp[type][indx], cnt[type][indx], bw_ctl,
2153 sz[type][indx], sz1, mp);
2154 continue;
2155 }
2156
2157 /*
2158 * By now, the fastpath requirements ensure that direct access
2159 * to the L3/L4 headers will fall safely within the mblk.
2160 */
2161 const ipha_t *ipha = (ipha_t *)(mp->b_rptr + meoi.meoi_l2hlen);
2162 const ip6_t *ip6 = (ip6_t *)(mp->b_rptr + meoi.meoi_l2hlen);
2163 const uint32_t *ports = (uint32_t *)
2164 (mp->b_rptr + meoi.meoi_l2hlen + meoi.meoi_l3hlen);
2165
2166 /*
2167 * XXX-Sunay: We should hold srs_lock since ring_count
2168 * below can change. But if we are always called from
2169 * mac_rx_srs_drain and SRS_PROC is set, then we can
2170 * enforce that ring_count can't be changed i.e.
2171 * to change fanout type or ring count, the calling
2172 * thread needs to be behind SRS_PROC.
2173 */
2174 uint_t hash;
2175 switch (meoi.meoi_l4proto) {
2176 case IPPROTO_TCP:
2177 /*
2178 * Note that for ESP, we fanout on SPI and it is at the
2179 * same offset as the 2x16-bit ports. So it is clumped
2180 * along with TCP, UDP and SCTP.
2181 */
2182 if (meoi.meoi_l3proto == ETHERTYPE_IP) {
2183 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst,
2184 *ports);
2185 type = V4_TCP;
2186 }
2187 if (meoi.meoi_l3proto == ETHERTYPE_IPV6) {
2188 hash = HASH_ADDR6(ip6->ip6_src, ip6->ip6_dst,
2189 *ports);
2190 type = V6_TCP;
2191 }
2192 indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count);
2193 mp->b_rptr += meoi.meoi_l2hlen;
2194 break;
2195 case IPPROTO_UDP:
2196 case IPPROTO_SCTP:
2197 case IPPROTO_ESP:
2198 if (mac_fanout_type == MAC_FANOUT_DEFAULT) {
2199 if (meoi.meoi_l3proto == ETHERTYPE_IP) {
2200 hash = HASH_ADDR(ipha->ipha_src,
2201 ipha->ipha_dst, *ports);
2202 }
2203 if (meoi.meoi_l3proto == ETHERTYPE_IPV6) {
2204 hash = HASH_ADDR6(ip6->ip6_src,
2205 ip6->ip6_dst, *ports);
2206 }
2207 indx = COMPUTE_INDEX(hash,
2208 mac_srs->srs_udp_ring_count);
2209 } else {
2210 indx = mac_srs->srs_ind %
2211 mac_srs->srs_udp_ring_count;
2212 mac_srs->srs_ind++;
2213 }
2214 type = (meoi.meoi_l3proto == ETHERTYPE_IPV6) ?
2215 V6_UDP : V4_UDP;
2216 mp->b_rptr += meoi.meoi_l2hlen;
2217 break;
2218 default:
2219 indx = 0;
2220 type = OTH;
2221 }
2222
2223 DTRACE_PROBE4(rx__fanout, mblk_t *, mp,
2224 mac_ether_offload_info_t *, &meoi, mac_soft_ring_set_t *,
2225 mac_srs, pkt_type_t, type);
2226 FANOUT_ENQUEUE_MP(headmp[type][indx], tailmp[type][indx],
2227 cnt[type][indx], bw_ctl, sz[type][indx], sz1, mp);
2228 }
2229
2230 for (pkt_type_t type = V4_TCP; type < UNDEF; type++) {
2231 for (int i = 0; i < fanout_cnt; i++) {
2232 if (headmp[type][i] != NULL) {
2233 mac_soft_ring_t *softring;
2234
2235 ASSERT(tailmp[type][i]->b_next == NULL);
2236 switch (type) {
2237 case V4_TCP:
2238 softring =
2239 mac_srs->srs_tcp_soft_rings[i];
2240 break;
2241 case V6_TCP:
2242 softring =
2243 mac_srs->srs_tcp6_soft_rings[i];
2244 break;
2245 case V4_UDP:
2246 softring =
2247 mac_srs->srs_udp_soft_rings[i];
2248 break;
2249 case V6_UDP:
2250 softring =
2251 mac_srs->srs_udp6_soft_rings[i];
2252 break;
2253 case OTH:
2254 softring =
2255 mac_srs->srs_oth_soft_rings[i];
2256 break;
2257 }
2258 mac_rx_soft_ring_process(mcip,
2259 softring, headmp[type][i], tailmp[type][i],
2260 cnt[type][i], sz[type][i]);
2261 }
2262 }
2263 }
2264 }
2265
2266 #define SRS_BYTES_TO_PICKUP 150000
2267 ssize_t max_bytes_to_pickup = SRS_BYTES_TO_PICKUP;
2268
2269 /*
2270 * mac_rx_srs_poll_ring
2271 *
2272 * This SRS Poll thread uses this routine to poll the underlying hardware
2273 * Rx ring to get a chain of packets. It can inline process that chain
2274 * if mac_latency_optimize is set (default) or signal the SRS worker thread
2275 * to do the remaining processing.
2276 *
2277 * Since packets come in the system via interrupt or poll path, we also
2278 * update the stats and deal with promiscous clients here.
2279 */
2280 void
mac_rx_srs_poll_ring(mac_soft_ring_set_t * mac_srs)2281 mac_rx_srs_poll_ring(mac_soft_ring_set_t *mac_srs)
2282 {
2283 kmutex_t *lock = &mac_srs->srs_lock;
2284 kcondvar_t *async = &mac_srs->srs_cv;
2285 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2286 mblk_t *head, *tail, *mp;
2287 callb_cpr_t cprinfo;
2288 ssize_t bytes_to_pickup;
2289 size_t sz;
2290 int count;
2291 mac_client_impl_t *smcip;
2292
2293 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "mac_srs_poll");
2294 mutex_enter(lock);
2295
2296 start:
2297 for (;;) {
2298 if (mac_srs->srs_state & SRS_PAUSE)
2299 goto done;
2300
2301 CALLB_CPR_SAFE_BEGIN(&cprinfo);
2302 cv_wait(async, lock);
2303 CALLB_CPR_SAFE_END(&cprinfo, lock);
2304
2305 if (mac_srs->srs_state & SRS_PAUSE)
2306 goto done;
2307
2308 check_again:
2309 if (mac_srs->srs_type & SRST_BW_CONTROL) {
2310 /*
2311 * We pick as many bytes as we are allowed to queue.
2312 * Its possible that we will exceed the total
2313 * packets queued in case this SRS is part of the
2314 * Rx ring group since > 1 poll thread can be pulling
2315 * upto the max allowed packets at the same time
2316 * but that should be OK.
2317 */
2318 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2319 bytes_to_pickup =
2320 mac_srs->srs_bw->mac_bw_drop_threshold -
2321 mac_srs->srs_bw->mac_bw_sz;
2322 /*
2323 * We shouldn't have been signalled if we
2324 * have 0 or less bytes to pick but since
2325 * some of the bytes accounting is driver
2326 * dependant, we do the safety check.
2327 */
2328 if (bytes_to_pickup < 0)
2329 bytes_to_pickup = 0;
2330 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2331 } else {
2332 /*
2333 * ToDO: Need to change the polling API
2334 * to add a packet count and a flag which
2335 * tells the driver whether we want packets
2336 * based on a count, or bytes, or all the
2337 * packets queued in the driver/HW. This
2338 * way, we never have to check the limits
2339 * on poll path. We truly let only as many
2340 * packets enter the system as we are willing
2341 * to process or queue.
2342 *
2343 * Something along the lines of
2344 * pkts_to_pickup = mac_soft_ring_max_q_cnt -
2345 * mac_srs->srs_poll_pkt_cnt
2346 */
2347
2348 /*
2349 * Since we are not doing B/W control, pick
2350 * as many packets as allowed.
2351 */
2352 bytes_to_pickup = max_bytes_to_pickup;
2353 }
2354
2355 /* Poll the underlying Hardware */
2356 mutex_exit(lock);
2357 head = MAC_HWRING_POLL(mac_srs->srs_ring, (int)bytes_to_pickup);
2358 mutex_enter(lock);
2359
2360 ASSERT((mac_srs->srs_state & SRS_POLL_THR_OWNER) ==
2361 SRS_POLL_THR_OWNER);
2362
2363 mp = tail = head;
2364 count = 0;
2365 sz = 0;
2366 while (mp != NULL) {
2367 tail = mp;
2368 sz += msgdsize(mp);
2369 mp = mp->b_next;
2370 count++;
2371 }
2372
2373 if (head != NULL) {
2374 tail->b_next = NULL;
2375 smcip = mac_srs->srs_mcip;
2376
2377 SRS_RX_STAT_UPDATE(mac_srs, pollbytes, sz);
2378 SRS_RX_STAT_UPDATE(mac_srs, pollcnt, count);
2379
2380 /*
2381 * If there are any promiscuous mode callbacks
2382 * defined for this MAC client, pass them a copy
2383 * if appropriate and also update the counters.
2384 */
2385 if (smcip != NULL) {
2386 if (smcip->mci_mip->mi_promisc_list != NULL) {
2387 mutex_exit(lock);
2388 mac_promisc_dispatch(smcip->mci_mip,
2389 head, NULL, B_FALSE);
2390 mutex_enter(lock);
2391 }
2392 }
2393 if (mac_srs->srs_type & SRST_BW_CONTROL) {
2394 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2395 mac_srs->srs_bw->mac_bw_polled += sz;
2396 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2397 }
2398 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail,
2399 count, sz);
2400 if (count <= 10)
2401 srs_rx->sr_stat.mrs_chaincntundr10++;
2402 else if (count > 10 && count <= 50)
2403 srs_rx->sr_stat.mrs_chaincnt10to50++;
2404 else
2405 srs_rx->sr_stat.mrs_chaincntover50++;
2406 }
2407
2408 /*
2409 * We are guaranteed that SRS_PROC will be set if we
2410 * are here. Also, poll thread gets to run only if
2411 * the drain was being done by a worker thread although
2412 * its possible that worker thread is still running
2413 * and poll thread was sent down to keep the pipeline
2414 * going instead of doing a complete drain and then
2415 * trying to poll the NIC.
2416 *
2417 * So we need to check SRS_WORKER flag to make sure
2418 * that the worker thread is not processing the queue
2419 * in parallel to us. The flags and conditions are
2420 * protected by the srs_lock to prevent any race. We
2421 * ensure that we don't drop the srs_lock from now
2422 * till the end and similarly we don't drop the srs_lock
2423 * in mac_rx_srs_drain() till similar condition check
2424 * are complete. The mac_rx_srs_drain() needs to ensure
2425 * that SRS_WORKER flag remains set as long as its
2426 * processing the queue.
2427 */
2428 if (!(mac_srs->srs_state & SRS_WORKER) &&
2429 (mac_srs->srs_first != NULL)) {
2430 /*
2431 * We have packets to process and worker thread
2432 * is not running. Check to see if poll thread is
2433 * allowed to process.
2434 */
2435 if (mac_srs->srs_state & SRS_LATENCY_OPT) {
2436 mac_srs->srs_drain_func(mac_srs, SRS_POLL_PROC);
2437 if (!(mac_srs->srs_state & SRS_PAUSE) &&
2438 srs_rx->sr_poll_pkt_cnt <=
2439 srs_rx->sr_lowat) {
2440 srs_rx->sr_poll_again++;
2441 goto check_again;
2442 }
2443 /*
2444 * We are already above low water mark
2445 * so stay in the polling mode but no
2446 * need to poll. Once we dip below
2447 * the polling threshold, the processing
2448 * thread (soft ring) will signal us
2449 * to poll again (MAC_UPDATE_SRS_COUNT)
2450 */
2451 srs_rx->sr_poll_drain_no_poll++;
2452 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS);
2453 /*
2454 * In B/W control case, its possible
2455 * that the backlog built up due to
2456 * B/W limit being reached and packets
2457 * are queued only in SRS. In this case,
2458 * we should schedule worker thread
2459 * since no one else will wake us up.
2460 */
2461 if ((mac_srs->srs_type & SRST_BW_CONTROL) &&
2462 (mac_srs->srs_tid == NULL)) {
2463 mac_srs->srs_tid =
2464 timeout(mac_srs_fire, mac_srs, 1);
2465 srs_rx->sr_poll_worker_wakeup++;
2466 }
2467 } else {
2468 /*
2469 * Wakeup the worker thread for more processing.
2470 * We optimize for throughput in this case.
2471 */
2472 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS);
2473 MAC_SRS_WORKER_WAKEUP(mac_srs);
2474 srs_rx->sr_poll_sig_worker++;
2475 }
2476 } else if ((mac_srs->srs_first == NULL) &&
2477 !(mac_srs->srs_state & SRS_WORKER)) {
2478 /*
2479 * There is nothing queued in SRS and
2480 * no worker thread running. Plus we
2481 * didn't get anything from the H/W
2482 * as well (head == NULL);
2483 */
2484 ASSERT(head == NULL);
2485 mac_srs->srs_state &=
2486 ~(SRS_PROC|SRS_GET_PKTS);
2487
2488 /*
2489 * If we have a packets in soft ring, don't allow
2490 * more packets to come into this SRS by keeping the
2491 * interrupts off but not polling the H/W. The
2492 * poll thread will get signaled as soon as
2493 * srs_poll_pkt_cnt dips below poll threshold.
2494 */
2495 if (srs_rx->sr_poll_pkt_cnt == 0) {
2496 srs_rx->sr_poll_intr_enable++;
2497 MAC_SRS_POLLING_OFF(mac_srs);
2498 } else {
2499 /*
2500 * We know nothing is queued in SRS
2501 * since we are here after checking
2502 * srs_first is NULL. The backlog
2503 * is entirely due to packets queued
2504 * in Soft ring which will wake us up
2505 * and get the interface out of polling
2506 * mode once the backlog dips below
2507 * sr_poll_thres.
2508 */
2509 srs_rx->sr_poll_no_poll++;
2510 }
2511 } else {
2512 /*
2513 * Worker thread is already running.
2514 * Nothing much to do. If the polling
2515 * was enabled, worker thread will deal
2516 * with that.
2517 */
2518 mac_srs->srs_state &= ~SRS_GET_PKTS;
2519 srs_rx->sr_poll_goto_sleep++;
2520 }
2521 }
2522 done:
2523 mac_srs->srs_state |= SRS_POLL_THR_QUIESCED;
2524 cv_signal(&mac_srs->srs_async);
2525 /*
2526 * If this is a temporary quiesce then wait for the restart signal
2527 * from the srs worker. Then clear the flags and signal the srs worker
2528 * to ensure a positive handshake and go back to start.
2529 */
2530 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_POLL_THR_RESTART)))
2531 cv_wait(async, lock);
2532 if (mac_srs->srs_state & SRS_POLL_THR_RESTART) {
2533 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED));
2534 mac_srs->srs_state &=
2535 ~(SRS_POLL_THR_QUIESCED | SRS_POLL_THR_RESTART);
2536 cv_signal(&mac_srs->srs_async);
2537 goto start;
2538 } else {
2539 mac_srs->srs_state |= SRS_POLL_THR_EXITED;
2540 cv_signal(&mac_srs->srs_async);
2541 CALLB_CPR_EXIT(&cprinfo);
2542 thread_exit();
2543 }
2544 }
2545
2546 /*
2547 * mac_srs_pick_chain
2548 *
2549 * In Bandwidth control case, checks how many packets can be processed
2550 * and return them in a sub chain.
2551 */
2552 static mblk_t *
mac_srs_pick_chain(mac_soft_ring_set_t * mac_srs,mblk_t ** chain_tail,size_t * chain_sz,int * chain_cnt)2553 mac_srs_pick_chain(mac_soft_ring_set_t *mac_srs, mblk_t **chain_tail,
2554 size_t *chain_sz, int *chain_cnt)
2555 {
2556 mblk_t *head = NULL;
2557 mblk_t *tail = NULL;
2558 size_t sz;
2559 size_t tsz = 0;
2560 int cnt = 0;
2561 mblk_t *mp;
2562
2563 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2564 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2565 if (((mac_srs->srs_bw->mac_bw_used + mac_srs->srs_size) <=
2566 mac_srs->srs_bw->mac_bw_limit) ||
2567 (mac_srs->srs_bw->mac_bw_limit == 0)) {
2568 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2569 head = mac_srs->srs_first;
2570 mac_srs->srs_first = NULL;
2571 *chain_tail = mac_srs->srs_last;
2572 mac_srs->srs_last = NULL;
2573 *chain_sz = mac_srs->srs_size;
2574 *chain_cnt = mac_srs->srs_count;
2575 mac_srs->srs_count = 0;
2576 mac_srs->srs_size = 0;
2577 return (head);
2578 }
2579
2580 /*
2581 * Can't clear the entire backlog.
2582 * Need to find how many packets to pick
2583 */
2584 ASSERT(MUTEX_HELD(&mac_srs->srs_bw->mac_bw_lock));
2585 while ((mp = mac_srs->srs_first) != NULL) {
2586 sz = msgdsize(mp);
2587 if ((tsz + sz + mac_srs->srs_bw->mac_bw_used) >
2588 mac_srs->srs_bw->mac_bw_limit) {
2589 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED))
2590 mac_srs->srs_bw->mac_bw_state |=
2591 SRS_BW_ENFORCED;
2592 break;
2593 }
2594
2595 /*
2596 * The _size & cnt is decremented from the softrings
2597 * when they send up the packet for polling to work
2598 * properly.
2599 */
2600 tsz += sz;
2601 cnt++;
2602 mac_srs->srs_count--;
2603 mac_srs->srs_size -= sz;
2604 if (tail != NULL)
2605 tail->b_next = mp;
2606 else
2607 head = mp;
2608 tail = mp;
2609 mac_srs->srs_first = mac_srs->srs_first->b_next;
2610 }
2611 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2612 if (mac_srs->srs_first == NULL)
2613 mac_srs->srs_last = NULL;
2614
2615 if (tail != NULL)
2616 tail->b_next = NULL;
2617 *chain_tail = tail;
2618 *chain_cnt = cnt;
2619 *chain_sz = tsz;
2620
2621 return (head);
2622 }
2623
2624 /*
2625 * mac_rx_srs_drain
2626 *
2627 * The SRS drain routine. Gets to run to clear the queue. Any thread
2628 * (worker, interrupt, poll) can call this based on processing model.
2629 * The first thing we do is disable interrupts if possible and then
2630 * drain the queue. we also try to poll the underlying hardware if
2631 * there is a dedicated hardware Rx ring assigned to this SRS.
2632 *
2633 * There is a equivalent drain routine in bandwidth control mode
2634 * mac_rx_srs_drain_bw. There is some code duplication between the two
2635 * routines but they are highly performance sensitive and are easier
2636 * to read/debug if they stay separate. Any code changes here might
2637 * also apply to mac_rx_srs_drain_bw as well.
2638 */
2639 void
mac_rx_srs_drain(mac_soft_ring_set_t * mac_srs,uint_t proc_type)2640 mac_rx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
2641 {
2642 mblk_t *head;
2643 mblk_t *tail;
2644 timeout_id_t tid;
2645 int cnt = 0;
2646 mac_client_impl_t *mcip = mac_srs->srs_mcip;
2647 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2648
2649 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2650 ASSERT(!(mac_srs->srs_type & SRST_BW_CONTROL));
2651
2652 /* If we are blanked i.e. can't do upcalls, then we are done */
2653 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) {
2654 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) ||
2655 (mac_srs->srs_state & SRS_PAUSE));
2656 goto out;
2657 }
2658
2659 if (mac_srs->srs_first == NULL)
2660 goto out;
2661
2662 if (!(mac_srs->srs_state & SRS_LATENCY_OPT) &&
2663 (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)) {
2664 /*
2665 * In the normal case, the SRS worker thread does no
2666 * work and we wait for a backlog to build up before
2667 * we switch into polling mode. In case we are
2668 * optimizing for throughput, we use the worker thread
2669 * as well. The goal is to let worker thread process
2670 * the queue and poll thread to feed packets into
2671 * the queue. As such, we should signal the poll
2672 * thread to try and get more packets.
2673 *
2674 * We could have pulled this check in the POLL_RING
2675 * macro itself but keeping it explicit here makes
2676 * the architecture more human understandable.
2677 */
2678 MAC_SRS_POLL_RING(mac_srs);
2679 }
2680
2681 again:
2682 head = mac_srs->srs_first;
2683 mac_srs->srs_first = NULL;
2684 tail = mac_srs->srs_last;
2685 mac_srs->srs_last = NULL;
2686 cnt = mac_srs->srs_count;
2687 mac_srs->srs_count = 0;
2688
2689 ASSERT(head != NULL);
2690 ASSERT(tail != NULL);
2691
2692 if ((tid = mac_srs->srs_tid) != NULL)
2693 mac_srs->srs_tid = NULL;
2694
2695 mac_srs->srs_state |= (SRS_PROC|proc_type);
2696
2697 /*
2698 * mcip is NULL for broadcast and multicast flows. The promisc
2699 * callbacks for broadcast and multicast packets are delivered from
2700 * mac_rx() and we don't need to worry about that case in this path
2701 */
2702 if (mcip != NULL) {
2703 if (mcip->mci_promisc_list != NULL) {
2704 mutex_exit(&mac_srs->srs_lock);
2705 mac_promisc_client_dispatch(mcip, head);
2706 mutex_enter(&mac_srs->srs_lock);
2707 }
2708 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) {
2709 mutex_exit(&mac_srs->srs_lock);
2710 mac_protect_intercept_dynamic(mcip, head);
2711 mutex_enter(&mac_srs->srs_lock);
2712 }
2713 }
2714
2715 /*
2716 * Check if SRS itself is doing the processing. This direct
2717 * path applies only when subflows are present.
2718 */
2719 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) {
2720 mac_direct_rx_t proc;
2721 void *arg1;
2722 mac_resource_handle_t arg2;
2723
2724 /*
2725 * This is the case when a Rx is directly
2726 * assigned and we have a fully classified
2727 * protocol chain. We can deal with it in
2728 * one shot.
2729 */
2730 proc = srs_rx->sr_func;
2731 arg1 = srs_rx->sr_arg1;
2732 arg2 = srs_rx->sr_arg2;
2733
2734 mac_srs->srs_state |= SRS_CLIENT_PROC;
2735 mutex_exit(&mac_srs->srs_lock);
2736 if (tid != NULL) {
2737 (void) untimeout(tid);
2738 tid = NULL;
2739 }
2740
2741 proc(arg1, arg2, head, NULL);
2742 /*
2743 * Decrement the size and count here itelf
2744 * since the packet has been processed.
2745 */
2746 mutex_enter(&mac_srs->srs_lock);
2747 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
2748 if (mac_srs->srs_state & SRS_CLIENT_WAIT)
2749 cv_signal(&mac_srs->srs_client_cv);
2750 mac_srs->srs_state &= ~SRS_CLIENT_PROC;
2751 } else {
2752 /* Some kind of softrings based fanout is required */
2753 mutex_exit(&mac_srs->srs_lock);
2754 if (tid != NULL) {
2755 (void) untimeout(tid);
2756 tid = NULL;
2757 }
2758
2759 /*
2760 * Since the fanout routines can deal with chains,
2761 * shoot the entire chain up.
2762 */
2763 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP)
2764 mac_rx_srs_fanout(mac_srs, head);
2765 else
2766 mac_rx_srs_proto_fanout(mac_srs, head);
2767 mutex_enter(&mac_srs->srs_lock);
2768 }
2769
2770 if (!(mac_srs->srs_state & (SRS_BLANK|SRS_PAUSE)) &&
2771 (mac_srs->srs_first != NULL)) {
2772 /*
2773 * More packets arrived while we were clearing the
2774 * SRS. This can be possible because of one of
2775 * three conditions below:
2776 * 1) The driver is using multiple worker threads
2777 * to send the packets to us.
2778 * 2) The driver has a race in switching
2779 * between interrupt and polling mode or
2780 * 3) Packets are arriving in this SRS via the
2781 * S/W classification as well.
2782 *
2783 * We should switch to polling mode and see if we
2784 * need to send the poll thread down. Also, signal
2785 * the worker thread to process whats just arrived.
2786 */
2787 MAC_SRS_POLLING_ON(mac_srs);
2788 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) {
2789 srs_rx->sr_drain_poll_sig++;
2790 MAC_SRS_POLL_RING(mac_srs);
2791 }
2792
2793 /*
2794 * If we didn't signal the poll thread, we need
2795 * to deal with the pending packets ourselves.
2796 */
2797 if (proc_type == SRS_WORKER) {
2798 srs_rx->sr_drain_again++;
2799 goto again;
2800 } else {
2801 srs_rx->sr_drain_worker_sig++;
2802 cv_signal(&mac_srs->srs_async);
2803 }
2804 }
2805
2806 out:
2807 if (mac_srs->srs_state & SRS_GET_PKTS) {
2808 /*
2809 * Poll thread is already running. Leave the
2810 * SRS_RPOC set and hand over the control to
2811 * poll thread.
2812 */
2813 mac_srs->srs_state &= ~proc_type;
2814 srs_rx->sr_drain_poll_running++;
2815 return;
2816 }
2817
2818 /*
2819 * Even if there are no packets queued in SRS, we
2820 * need to make sure that the shared counter is
2821 * clear and any associated softrings have cleared
2822 * all the backlog. Otherwise, leave the interface
2823 * in polling mode and the poll thread will get
2824 * signalled once the count goes down to zero.
2825 *
2826 * If someone is already draining the queue (SRS_PROC is
2827 * set) when the srs_poll_pkt_cnt goes down to zero,
2828 * then it means that drain is already running and we
2829 * will turn off polling at that time if there is
2830 * no backlog.
2831 *
2832 * As long as there are packets queued either
2833 * in soft ring set or its soft rings, we will leave
2834 * the interface in polling mode (even if the drain
2835 * was done being the interrupt thread). We signal
2836 * the poll thread as well if we have dipped below
2837 * low water mark.
2838 *
2839 * NOTE: We can't use the MAC_SRS_POLLING_ON macro
2840 * since that turn polling on only for worker thread.
2841 * Its not worth turning polling on for interrupt
2842 * thread (since NIC will not issue another interrupt)
2843 * unless a backlog builds up.
2844 */
2845 if ((srs_rx->sr_poll_pkt_cnt > 0) &&
2846 (mac_srs->srs_state & SRS_POLLING_CAPAB)) {
2847 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
2848 srs_rx->sr_drain_keep_polling++;
2849 MAC_SRS_POLLING_ON(mac_srs);
2850 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)
2851 MAC_SRS_POLL_RING(mac_srs);
2852 return;
2853 }
2854
2855 /* Nothing else to do. Get out of poll mode */
2856 MAC_SRS_POLLING_OFF(mac_srs);
2857 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
2858 srs_rx->sr_drain_finish_intr++;
2859 }
2860
2861 /*
2862 * mac_rx_srs_drain_bw
2863 *
2864 * The SRS BW drain routine. Gets to run to clear the queue. Any thread
2865 * (worker, interrupt, poll) can call this based on processing model.
2866 * The first thing we do is disable interrupts if possible and then
2867 * drain the queue. we also try to poll the underlying hardware if
2868 * there is a dedicated hardware Rx ring assigned to this SRS.
2869 *
2870 * There is a equivalent drain routine in non bandwidth control mode
2871 * mac_rx_srs_drain. There is some code duplication between the two
2872 * routines but they are highly performance sensitive and are easier
2873 * to read/debug if they stay separate. Any code changes here might
2874 * also apply to mac_rx_srs_drain as well.
2875 */
2876 void
mac_rx_srs_drain_bw(mac_soft_ring_set_t * mac_srs,uint_t proc_type)2877 mac_rx_srs_drain_bw(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
2878 {
2879 mblk_t *head;
2880 mblk_t *tail;
2881 timeout_id_t tid;
2882 size_t sz = 0;
2883 int cnt = 0;
2884 mac_client_impl_t *mcip = mac_srs->srs_mcip;
2885 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2886 clock_t now;
2887
2888 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2889 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL);
2890 again:
2891 /* Check if we are doing B/W control */
2892 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2893 now = ddi_get_lbolt();
2894 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
2895 mac_srs->srs_bw->mac_bw_curr_time = now;
2896 mac_srs->srs_bw->mac_bw_used = 0;
2897 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)
2898 mac_srs->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED;
2899 } else if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) {
2900 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2901 goto done;
2902 } else if (mac_srs->srs_bw->mac_bw_used >
2903 mac_srs->srs_bw->mac_bw_limit) {
2904 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
2905 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2906 goto done;
2907 }
2908 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2909
2910 /* If we are blanked i.e. can't do upcalls, then we are done */
2911 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) {
2912 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) ||
2913 (mac_srs->srs_state & SRS_PAUSE));
2914 goto done;
2915 }
2916
2917 sz = 0;
2918 cnt = 0;
2919 if ((head = mac_srs_pick_chain(mac_srs, &tail, &sz, &cnt)) == NULL) {
2920 /*
2921 * We couldn't pick up a single packet.
2922 */
2923 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2924 if ((mac_srs->srs_bw->mac_bw_used == 0) &&
2925 (mac_srs->srs_size != 0) &&
2926 !(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
2927 /*
2928 * Seems like configured B/W doesn't
2929 * even allow processing of 1 packet
2930 * per tick.
2931 *
2932 * XXX: raise the limit to processing
2933 * at least 1 packet per tick.
2934 */
2935 mac_srs->srs_bw->mac_bw_limit +=
2936 mac_srs->srs_bw->mac_bw_limit;
2937 mac_srs->srs_bw->mac_bw_drop_threshold +=
2938 mac_srs->srs_bw->mac_bw_drop_threshold;
2939 cmn_err(CE_NOTE, "mac_rx_srs_drain: srs(%p) "
2940 "raised B/W limit to %d since not even a "
2941 "single packet can be processed per "
2942 "tick %d\n", (void *)mac_srs,
2943 (int)mac_srs->srs_bw->mac_bw_limit,
2944 (int)msgdsize(mac_srs->srs_first));
2945 }
2946 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2947 goto done;
2948 }
2949
2950 ASSERT(head != NULL);
2951 ASSERT(tail != NULL);
2952
2953 /* zero bandwidth: drop all and return to interrupt mode */
2954 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2955 if (mac_srs->srs_bw->mac_bw_limit == 0) {
2956 srs_rx->sr_stat.mrs_sdrops += cnt;
2957 ASSERT(mac_srs->srs_bw->mac_bw_sz >= sz);
2958 mac_srs->srs_bw->mac_bw_sz -= sz;
2959 mac_srs->srs_bw->mac_bw_drop_bytes += sz;
2960 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2961 mac_drop_chain(head, "Rx no bandwidth");
2962 goto leave_poll;
2963 } else {
2964 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2965 }
2966
2967 if ((tid = mac_srs->srs_tid) != NULL)
2968 mac_srs->srs_tid = NULL;
2969
2970 mac_srs->srs_state |= (SRS_PROC|proc_type);
2971 MAC_SRS_WORKER_POLLING_ON(mac_srs);
2972
2973 /*
2974 * mcip is NULL for broadcast and multicast flows. The promisc
2975 * callbacks for broadcast and multicast packets are delivered from
2976 * mac_rx() and we don't need to worry about that case in this path
2977 */
2978 if (mcip != NULL) {
2979 if (mcip->mci_promisc_list != NULL) {
2980 mutex_exit(&mac_srs->srs_lock);
2981 mac_promisc_client_dispatch(mcip, head);
2982 mutex_enter(&mac_srs->srs_lock);
2983 }
2984 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) {
2985 mutex_exit(&mac_srs->srs_lock);
2986 mac_protect_intercept_dynamic(mcip, head);
2987 mutex_enter(&mac_srs->srs_lock);
2988 }
2989 }
2990
2991 /*
2992 * Check if SRS itself is doing the processing
2993 * This direct path does not apply when subflows are present. In this
2994 * case, packets need to be dispatched to a soft ring according to the
2995 * flow's bandwidth and other resources contraints.
2996 */
2997 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) {
2998 mac_direct_rx_t proc;
2999 void *arg1;
3000 mac_resource_handle_t arg2;
3001
3002 /*
3003 * This is the case when a Rx is directly
3004 * assigned and we have a fully classified
3005 * protocol chain. We can deal with it in
3006 * one shot.
3007 */
3008 proc = srs_rx->sr_func;
3009 arg1 = srs_rx->sr_arg1;
3010 arg2 = srs_rx->sr_arg2;
3011
3012 mac_srs->srs_state |= SRS_CLIENT_PROC;
3013 mutex_exit(&mac_srs->srs_lock);
3014 if (tid != NULL) {
3015 (void) untimeout(tid);
3016 tid = NULL;
3017 }
3018
3019 proc(arg1, arg2, head, NULL);
3020 /*
3021 * Decrement the size and count here itelf
3022 * since the packet has been processed.
3023 */
3024 mutex_enter(&mac_srs->srs_lock);
3025 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
3026 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz);
3027
3028 if (mac_srs->srs_state & SRS_CLIENT_WAIT)
3029 cv_signal(&mac_srs->srs_client_cv);
3030 mac_srs->srs_state &= ~SRS_CLIENT_PROC;
3031 } else {
3032 /* Some kind of softrings based fanout is required */
3033 mutex_exit(&mac_srs->srs_lock);
3034 if (tid != NULL) {
3035 (void) untimeout(tid);
3036 tid = NULL;
3037 }
3038
3039 /*
3040 * Since the fanout routines can deal with chains,
3041 * shoot the entire chain up.
3042 */
3043 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP)
3044 mac_rx_srs_fanout(mac_srs, head);
3045 else
3046 mac_rx_srs_proto_fanout(mac_srs, head);
3047 mutex_enter(&mac_srs->srs_lock);
3048 }
3049
3050 /*
3051 * Send the poll thread to pick up any packets arrived
3052 * so far. This also serves as the last check in case
3053 * nothing else is queued in the SRS. The poll thread
3054 * is signalled only in the case the drain was done
3055 * by the worker thread and SRS_WORKER is set. The
3056 * worker thread can run in parallel as long as the
3057 * SRS_WORKER flag is set. We we have nothing else to
3058 * process, we can exit while leaving SRS_PROC set
3059 * which gives the poll thread control to process and
3060 * cleanup once it returns from the NIC.
3061 *
3062 * If we have nothing else to process, we need to
3063 * ensure that we keep holding the srs_lock till
3064 * all the checks below are done and control is
3065 * handed to the poll thread if it was running.
3066 */
3067 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
3068 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
3069 if (mac_srs->srs_first != NULL) {
3070 if (proc_type == SRS_WORKER) {
3071 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3072 if (srs_rx->sr_poll_pkt_cnt <=
3073 srs_rx->sr_lowat)
3074 MAC_SRS_POLL_RING(mac_srs);
3075 goto again;
3076 } else {
3077 cv_signal(&mac_srs->srs_async);
3078 }
3079 }
3080 }
3081 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3082
3083 done:
3084
3085 if (mac_srs->srs_state & SRS_GET_PKTS) {
3086 /*
3087 * Poll thread is already running. Leave the
3088 * SRS_RPOC set and hand over the control to
3089 * poll thread.
3090 */
3091 mac_srs->srs_state &= ~proc_type;
3092 return;
3093 }
3094
3095 /*
3096 * If we can't process packets because we have exceeded
3097 * B/W limit for this tick, just set the timeout
3098 * and leave.
3099 *
3100 * Even if there are no packets queued in SRS, we
3101 * need to make sure that the shared counter is
3102 * clear and any associated softrings have cleared
3103 * all the backlog. Otherwise, leave the interface
3104 * in polling mode and the poll thread will get
3105 * signalled once the count goes down to zero.
3106 *
3107 * If someone is already draining the queue (SRS_PROC is
3108 * set) when the srs_poll_pkt_cnt goes down to zero,
3109 * then it means that drain is already running and we
3110 * will turn off polling at that time if there is
3111 * no backlog. As long as there are packets queued either
3112 * is soft ring set or its soft rings, we will leave
3113 * the interface in polling mode.
3114 */
3115 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
3116 if ((mac_srs->srs_state & SRS_POLLING_CAPAB) &&
3117 ((mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) ||
3118 (srs_rx->sr_poll_pkt_cnt > 0))) {
3119 MAC_SRS_POLLING_ON(mac_srs);
3120 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
3121 if ((mac_srs->srs_first != NULL) &&
3122 (mac_srs->srs_tid == NULL))
3123 mac_srs->srs_tid = timeout(mac_srs_fire,
3124 mac_srs, 1);
3125 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3126 return;
3127 }
3128 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3129
3130 leave_poll:
3131
3132 /* Nothing else to do. Get out of poll mode */
3133 MAC_SRS_POLLING_OFF(mac_srs);
3134 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
3135 }
3136
3137 /*
3138 * mac_srs_worker
3139 *
3140 * The SRS worker routine. Drains the queue when no one else is
3141 * processing it.
3142 */
3143 void
mac_srs_worker(mac_soft_ring_set_t * mac_srs)3144 mac_srs_worker(mac_soft_ring_set_t *mac_srs)
3145 {
3146 kmutex_t *lock = &mac_srs->srs_lock;
3147 kcondvar_t *async = &mac_srs->srs_async;
3148 callb_cpr_t cprinfo;
3149 boolean_t bw_ctl_flag;
3150
3151 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "srs_worker");
3152 mutex_enter(lock);
3153
3154 start:
3155 for (;;) {
3156 bw_ctl_flag = B_FALSE;
3157 if (mac_srs->srs_type & SRST_BW_CONTROL) {
3158 MAC_SRS_BW_LOCK(mac_srs);
3159 MAC_SRS_CHECK_BW_CONTROL(mac_srs);
3160 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)
3161 bw_ctl_flag = B_TRUE;
3162 MAC_SRS_BW_UNLOCK(mac_srs);
3163 }
3164 /*
3165 * The SRS_BW_ENFORCED flag may change since we have dropped
3166 * the mac_bw_lock. However the drain function can handle both
3167 * a drainable SRS or a bandwidth controlled SRS, and the
3168 * effect of scheduling a timeout is to wakeup the worker
3169 * thread which in turn will call the drain function. Since
3170 * we release the srs_lock atomically only in the cv_wait there
3171 * isn't a fear of waiting for ever.
3172 */
3173 while (((mac_srs->srs_state & SRS_PROC) ||
3174 (mac_srs->srs_first == NULL) || bw_ctl_flag ||
3175 (mac_srs->srs_state & SRS_TX_BLOCKED)) &&
3176 !(mac_srs->srs_state & SRS_PAUSE)) {
3177 /*
3178 * If we have packets queued and we are here
3179 * because B/W control is in place, we better
3180 * schedule the worker wakeup after 1 tick
3181 * to see if bandwidth control can be relaxed.
3182 */
3183 if (bw_ctl_flag && mac_srs->srs_tid == NULL) {
3184 /*
3185 * We need to ensure that a timer is already
3186 * scheduled or we force schedule one for
3187 * later so that we can continue processing
3188 * after this quanta is over.
3189 */
3190 mac_srs->srs_tid = timeout(mac_srs_fire,
3191 mac_srs, 1);
3192 }
3193 wait:
3194 CALLB_CPR_SAFE_BEGIN(&cprinfo);
3195 cv_wait(async, lock);
3196 CALLB_CPR_SAFE_END(&cprinfo, lock);
3197
3198 if (mac_srs->srs_state & SRS_PAUSE)
3199 goto done;
3200 if (mac_srs->srs_state & SRS_PROC)
3201 goto wait;
3202
3203 if (mac_srs->srs_first != NULL &&
3204 mac_srs->srs_type & SRST_BW_CONTROL) {
3205 MAC_SRS_BW_LOCK(mac_srs);
3206 if (mac_srs->srs_bw->mac_bw_state &
3207 SRS_BW_ENFORCED) {
3208 MAC_SRS_CHECK_BW_CONTROL(mac_srs);
3209 }
3210 bw_ctl_flag = mac_srs->srs_bw->mac_bw_state &
3211 SRS_BW_ENFORCED;
3212 MAC_SRS_BW_UNLOCK(mac_srs);
3213 }
3214 }
3215
3216 if (mac_srs->srs_state & SRS_PAUSE)
3217 goto done;
3218 mac_srs->srs_drain_func(mac_srs, SRS_WORKER);
3219 }
3220 done:
3221 /*
3222 * The Rx SRS quiesce logic first cuts off packet supply to the SRS
3223 * from both hard and soft classifications and waits for such threads
3224 * to finish before signaling the worker. So at this point the only
3225 * thread left that could be competing with the worker is the poll
3226 * thread. In the case of Tx, there shouldn't be any thread holding
3227 * SRS_PROC at this point.
3228 */
3229 if (!(mac_srs->srs_state & SRS_PROC)) {
3230 mac_srs->srs_state |= SRS_PROC;
3231 } else {
3232 ASSERT((mac_srs->srs_type & SRST_TX) == 0);
3233 /*
3234 * Poll thread still owns the SRS and is still running
3235 */
3236 ASSERT((mac_srs->srs_poll_thr == NULL) ||
3237 ((mac_srs->srs_state & SRS_POLL_THR_OWNER) ==
3238 SRS_POLL_THR_OWNER));
3239 }
3240 mac_srs_worker_quiesce(mac_srs);
3241 /*
3242 * Wait for the SRS_RESTART or SRS_CONDEMNED signal from the initiator
3243 * of the quiesce operation
3244 */
3245 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_RESTART)))
3246 cv_wait(&mac_srs->srs_async, &mac_srs->srs_lock);
3247
3248 if (mac_srs->srs_state & SRS_RESTART) {
3249 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED));
3250 mac_srs_worker_restart(mac_srs);
3251 mac_srs->srs_state &= ~SRS_PROC;
3252 goto start;
3253 }
3254
3255 if (!(mac_srs->srs_state & SRS_CONDEMNED_DONE))
3256 mac_srs_worker_quiesce(mac_srs);
3257
3258 mac_srs->srs_state &= ~SRS_PROC;
3259 /* The macro drops the srs_lock */
3260 CALLB_CPR_EXIT(&cprinfo);
3261 thread_exit();
3262 }
3263
3264 /*
3265 * mac_rx_srs_subflow_process
3266 *
3267 * Receive side routine called from interrupt path when there are
3268 * sub flows present on this SRS.
3269 */
3270 /* ARGSUSED */
3271 void
mac_rx_srs_subflow_process(void * arg,mac_resource_handle_t srs,mblk_t * mp_chain,boolean_t loopback)3272 mac_rx_srs_subflow_process(void *arg, mac_resource_handle_t srs,
3273 mblk_t *mp_chain, boolean_t loopback)
3274 {
3275 flow_entry_t *flent = NULL;
3276 flow_entry_t *prev_flent = NULL;
3277 mblk_t *mp = NULL;
3278 mblk_t *tail = NULL;
3279 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs;
3280 mac_client_impl_t *mcip;
3281
3282 mcip = mac_srs->srs_mcip;
3283 ASSERT(mcip != NULL);
3284
3285 /*
3286 * We need to determine the SRS for every packet
3287 * by walking the flow table, if we don't get any,
3288 * then we proceed using the SRS we came with.
3289 */
3290 mp = tail = mp_chain;
3291 while (mp != NULL) {
3292
3293 /*
3294 * We will increment the stats for the matching subflow.
3295 * when we get the bytes/pkt count for the classified packets
3296 * later in mac_rx_srs_process.
3297 */
3298 (void) mac_flow_lookup(mcip->mci_subflow_tab, mp,
3299 FLOW_INBOUND, &flent);
3300
3301 if (mp == mp_chain || flent == prev_flent) {
3302 if (prev_flent != NULL)
3303 FLOW_REFRELE(prev_flent);
3304 prev_flent = flent;
3305 flent = NULL;
3306 tail = mp;
3307 mp = mp->b_next;
3308 continue;
3309 }
3310 tail->b_next = NULL;
3311 /*
3312 * A null indicates, this is for the mac_srs itself.
3313 * XXX-venu : probably assert for fe_rx_srs_cnt == 0.
3314 */
3315 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) {
3316 mac_rx_srs_process(arg,
3317 (mac_resource_handle_t)mac_srs, mp_chain,
3318 loopback);
3319 } else {
3320 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1,
3321 prev_flent->fe_cb_arg2, mp_chain, loopback);
3322 FLOW_REFRELE(prev_flent);
3323 }
3324 prev_flent = flent;
3325 flent = NULL;
3326 mp_chain = mp;
3327 tail = mp;
3328 mp = mp->b_next;
3329 }
3330 /* Last chain */
3331 ASSERT(mp_chain != NULL);
3332 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) {
3333 mac_rx_srs_process(arg,
3334 (mac_resource_handle_t)mac_srs, mp_chain, loopback);
3335 } else {
3336 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1,
3337 prev_flent->fe_cb_arg2, mp_chain, loopback);
3338 FLOW_REFRELE(prev_flent);
3339 }
3340 }
3341
3342 /*
3343 * MAC SRS receive side routine. If the data is coming from the
3344 * network (i.e. from a NIC) then this is called in interrupt context.
3345 * If the data is coming from a local sender (e.g. mac_tx_send() or
3346 * bridge_forward()) then this is not called in interrupt context.
3347 *
3348 * loopback is set to force a context switch on the loopback
3349 * path between MAC clients.
3350 */
3351 /* ARGSUSED */
3352 void
mac_rx_srs_process(void * arg,mac_resource_handle_t srs,mblk_t * mp_chain,boolean_t loopback)3353 mac_rx_srs_process(void *arg, mac_resource_handle_t srs, mblk_t *mp_chain,
3354 boolean_t loopback)
3355 {
3356 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs;
3357 mblk_t *mp, *tail, *head;
3358 int count = 0;
3359 int count1;
3360 size_t sz = 0;
3361 size_t chain_sz, sz1;
3362 mac_bw_ctl_t *mac_bw;
3363 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
3364
3365 /*
3366 * Set the tail, count and sz. We set the sz irrespective
3367 * of whether we are doing B/W control or not for the
3368 * purpose of updating the stats.
3369 */
3370 mp = tail = mp_chain;
3371 while (mp != NULL) {
3372 tail = mp;
3373 count++;
3374 sz += msgdsize(mp);
3375 mp = mp->b_next;
3376 }
3377
3378 mutex_enter(&mac_srs->srs_lock);
3379
3380 if (loopback) {
3381 SRS_RX_STAT_UPDATE(mac_srs, lclbytes, sz);
3382 SRS_RX_STAT_UPDATE(mac_srs, lclcnt, count);
3383
3384 } else {
3385 SRS_RX_STAT_UPDATE(mac_srs, intrbytes, sz);
3386 SRS_RX_STAT_UPDATE(mac_srs, intrcnt, count);
3387 }
3388
3389 /*
3390 * If the SRS in already being processed; has been blanked;
3391 * can be processed by worker thread only; or the B/W limit
3392 * has been reached, then queue the chain and check if
3393 * worker thread needs to be awakend.
3394 */
3395 if (mac_srs->srs_type & SRST_BW_CONTROL) {
3396 mac_bw = mac_srs->srs_bw;
3397 ASSERT(mac_bw != NULL);
3398 mutex_enter(&mac_bw->mac_bw_lock);
3399 mac_bw->mac_bw_intr += sz;
3400 if (mac_bw->mac_bw_limit == 0) {
3401 /* zero bandwidth: drop all */
3402 srs_rx->sr_stat.mrs_sdrops += count;
3403 mac_bw->mac_bw_drop_bytes += sz;
3404 mutex_exit(&mac_bw->mac_bw_lock);
3405 mutex_exit(&mac_srs->srs_lock);
3406 mac_drop_chain(mp_chain, "Rx no bandwidth");
3407 return;
3408 } else {
3409 if ((mac_bw->mac_bw_sz + sz) <=
3410 mac_bw->mac_bw_drop_threshold) {
3411 mutex_exit(&mac_bw->mac_bw_lock);
3412 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain,
3413 tail, count, sz);
3414 } else {
3415 mp = mp_chain;
3416 chain_sz = 0;
3417 count1 = 0;
3418 tail = NULL;
3419 head = NULL;
3420 while (mp != NULL) {
3421 sz1 = msgdsize(mp);
3422 if (mac_bw->mac_bw_sz + chain_sz + sz1 >
3423 mac_bw->mac_bw_drop_threshold)
3424 break;
3425 chain_sz += sz1;
3426 count1++;
3427 tail = mp;
3428 mp = mp->b_next;
3429 }
3430 mutex_exit(&mac_bw->mac_bw_lock);
3431 if (tail != NULL) {
3432 head = tail->b_next;
3433 tail->b_next = NULL;
3434 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs,
3435 mp_chain, tail, count1, chain_sz);
3436 sz -= chain_sz;
3437 count -= count1;
3438 } else {
3439 /* Can't pick up any */
3440 head = mp_chain;
3441 }
3442 if (head != NULL) {
3443 /* Drop any packet over the threshold */
3444 srs_rx->sr_stat.mrs_sdrops += count;
3445 mutex_enter(&mac_bw->mac_bw_lock);
3446 mac_bw->mac_bw_drop_bytes += sz;
3447 mutex_exit(&mac_bw->mac_bw_lock);
3448 freemsgchain(head);
3449 }
3450 }
3451 MAC_SRS_WORKER_WAKEUP(mac_srs);
3452 mutex_exit(&mac_srs->srs_lock);
3453 return;
3454 }
3455 }
3456
3457 /*
3458 * If the total number of packets queued in the SRS and
3459 * its associated soft rings exceeds the max allowed,
3460 * then drop the chain. If we are polling capable, this
3461 * shouldn't be happening.
3462 */
3463 if (!(mac_srs->srs_type & SRST_BW_CONTROL) &&
3464 (srs_rx->sr_poll_pkt_cnt > srs_rx->sr_hiwat)) {
3465 mac_bw = mac_srs->srs_bw;
3466 srs_rx->sr_stat.mrs_sdrops += count;
3467 mutex_enter(&mac_bw->mac_bw_lock);
3468 mac_bw->mac_bw_drop_bytes += sz;
3469 mutex_exit(&mac_bw->mac_bw_lock);
3470 freemsgchain(mp_chain);
3471 mutex_exit(&mac_srs->srs_lock);
3472 return;
3473 }
3474
3475 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, tail, count, sz);
3476
3477 if (!(mac_srs->srs_state & SRS_PROC)) {
3478 /*
3479 * If we are coming via loopback, if we are not optimizing for
3480 * latency, or if our stack is running deep, we should signal
3481 * the worker thread.
3482 */
3483 if (loopback || !(mac_srs->srs_state & SRS_LATENCY_OPT)) {
3484 /*
3485 * For loopback, We need to let the worker take
3486 * over as we don't want to continue in the same
3487 * thread even if we can. This could lead to stack
3488 * overflows and may also end up using
3489 * resources (cpu) incorrectly.
3490 */
3491 cv_signal(&mac_srs->srs_async);
3492 } else if (STACK_BIAS + (uintptr_t)getfp() -
3493 (uintptr_t)curthread->t_stkbase < mac_rx_srs_stack_needed) {
3494 if (++mac_rx_srs_stack_toodeep == 0)
3495 mac_rx_srs_stack_toodeep = 1;
3496 cv_signal(&mac_srs->srs_async);
3497 } else {
3498 /*
3499 * Seems like no one is processing the SRS and
3500 * there is no backlog. We also inline process
3501 * our packet if its a single packet in non
3502 * latency optimized case (in latency optimized
3503 * case, we inline process chains of any size).
3504 */
3505 mac_srs->srs_drain_func(mac_srs, SRS_PROC_FAST);
3506 }
3507 }
3508 mutex_exit(&mac_srs->srs_lock);
3509 }
3510
3511 /* TX SIDE ROUTINES (RUNTIME) */
3512
3513 /*
3514 * mac_tx_srs_no_desc
3515 *
3516 * This routine is called by Tx single ring default mode
3517 * when Tx ring runs out of descs.
3518 */
3519 mac_tx_cookie_t
mac_tx_srs_no_desc(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uint16_t flag,mblk_t ** ret_mp)3520 mac_tx_srs_no_desc(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3521 uint16_t flag, mblk_t **ret_mp)
3522 {
3523 mac_tx_cookie_t cookie = 0;
3524 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3525 boolean_t wakeup_worker = B_TRUE;
3526 uint32_t tx_mode = srs_tx->st_mode;
3527 int cnt, sz;
3528 mblk_t *tail;
3529
3530 ASSERT(tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_BW);
3531 if (flag & MAC_DROP_ON_NO_DESC) {
3532 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie,
3533 "Tx no desc");
3534 } else {
3535 if (mac_srs->srs_first != NULL)
3536 wakeup_worker = B_FALSE;
3537 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3538 if (flag & MAC_TX_NO_ENQUEUE) {
3539 /*
3540 * If TX_QUEUED is not set, queue the
3541 * packet and let mac_tx_srs_drain()
3542 * set the TX_BLOCKED bit for the
3543 * reasons explained above. Otherwise,
3544 * return the mblks.
3545 */
3546 if (wakeup_worker) {
3547 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3548 mp_chain, tail, cnt, sz);
3549 } else {
3550 MAC_TX_SET_NO_ENQUEUE(mac_srs,
3551 mp_chain, ret_mp, cookie);
3552 }
3553 } else {
3554 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain,
3555 tail, cnt, sz, cookie);
3556 }
3557 if (wakeup_worker)
3558 cv_signal(&mac_srs->srs_async);
3559 }
3560 return (cookie);
3561 }
3562
3563 /*
3564 * mac_tx_srs_enqueue
3565 *
3566 * This routine is called when Tx SRS is operating in either serializer
3567 * or bandwidth mode. In serializer mode, a packet will get enqueued
3568 * when a thread cannot enter SRS exclusively. In bandwidth mode,
3569 * packets gets queued if allowed byte-count limit for a tick is
3570 * exceeded. The action that gets taken when MAC_DROP_ON_NO_DESC and
3571 * MAC_TX_NO_ENQUEUE is set is different than when operaing in either
3572 * the default mode or fanout mode. Here packets get dropped or
3573 * returned back to the caller only after hi-watermark worth of data
3574 * is queued.
3575 */
3576 static mac_tx_cookie_t
mac_tx_srs_enqueue(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uint16_t flag,uintptr_t fanout_hint,mblk_t ** ret_mp)3577 mac_tx_srs_enqueue(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3578 uint16_t flag, uintptr_t fanout_hint, mblk_t **ret_mp)
3579 {
3580 mac_tx_cookie_t cookie = 0;
3581 int cnt, sz;
3582 mblk_t *tail;
3583 boolean_t wakeup_worker = B_TRUE;
3584
3585 /*
3586 * Ignore fanout hint if we don't have multiple tx rings.
3587 */
3588 if (!MAC_TX_SOFT_RINGS(mac_srs))
3589 fanout_hint = 0;
3590
3591 if (mac_srs->srs_first != NULL)
3592 wakeup_worker = B_FALSE;
3593 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3594 if (flag & MAC_DROP_ON_NO_DESC) {
3595 if (mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) {
3596 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie,
3597 "Tx SRS hiwat");
3598 } else {
3599 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3600 mp_chain, tail, cnt, sz);
3601 }
3602 } else if (flag & MAC_TX_NO_ENQUEUE) {
3603 if ((mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) ||
3604 (mac_srs->srs_state & SRS_TX_WAKEUP_CLIENT)) {
3605 MAC_TX_SET_NO_ENQUEUE(mac_srs, mp_chain,
3606 ret_mp, cookie);
3607 } else {
3608 mp_chain->b_prev = (mblk_t *)fanout_hint;
3609 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3610 mp_chain, tail, cnt, sz);
3611 }
3612 } else {
3613 /*
3614 * If you are BW_ENFORCED, just enqueue the
3615 * packet. srs_worker will drain it at the
3616 * prescribed rate. Before enqueueing, save
3617 * the fanout hint.
3618 */
3619 mp_chain->b_prev = (mblk_t *)fanout_hint;
3620 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain,
3621 tail, cnt, sz, cookie);
3622 }
3623 if (wakeup_worker)
3624 cv_signal(&mac_srs->srs_async);
3625 return (cookie);
3626 }
3627
3628 /*
3629 * There are seven tx modes:
3630 *
3631 * 1) Default mode (SRS_TX_DEFAULT)
3632 * 2) Serialization mode (SRS_TX_SERIALIZE)
3633 * 3) Fanout mode (SRS_TX_FANOUT)
3634 * 4) Bandwdith mode (SRS_TX_BW)
3635 * 5) Fanout and Bandwidth mode (SRS_TX_BW_FANOUT)
3636 * 6) aggr Tx mode (SRS_TX_AGGR)
3637 * 7) aggr Tx bw mode (SRS_TX_BW_AGGR)
3638 *
3639 * The tx mode in which an SRS operates is decided in mac_tx_srs_setup()
3640 * based on the number of Tx rings requested for an SRS and whether
3641 * bandwidth control is requested or not.
3642 *
3643 * The default mode (i.e., no fanout/no bandwidth) is used when the
3644 * underlying NIC does not have Tx rings or just one Tx ring. In this mode,
3645 * the SRS acts as a pass-thru. Packets will go directly to mac_tx_send().
3646 * When the underlying Tx ring runs out of Tx descs, it starts queueing up
3647 * packets in SRS. When flow-control is relieved, the srs_worker drains
3648 * the queued packets and informs blocked clients to restart sending
3649 * packets.
3650 *
3651 * In the SRS_TX_SERIALIZE mode, all calls to mac_tx() are serialized. This
3652 * mode is used when the link has no Tx rings or only one Tx ring.
3653 *
3654 * In the SRS_TX_FANOUT mode, packets will be fanned out to multiple
3655 * Tx rings. Each Tx ring will have a soft ring associated with it.
3656 * These soft rings will be hung off the Tx SRS. Queueing if it happens
3657 * due to lack of Tx desc will be in individual soft ring (and not srs)
3658 * associated with Tx ring.
3659 *
3660 * In the TX_BW mode, tx srs will allow packets to go down to Tx ring
3661 * only if bw is available. Otherwise the packets will be queued in
3662 * SRS. If fanout to multiple Tx rings is configured, the packets will
3663 * be fanned out among the soft rings associated with the Tx rings.
3664 *
3665 * In SRS_TX_AGGR mode, mac_tx_aggr_mode() routine is called. This routine
3666 * invokes an aggr function, aggr_find_tx_ring(), to find a pseudo Tx ring
3667 * belonging to a port on which the packet has to be sent. Aggr will
3668 * always have a pseudo Tx ring associated with it even when it is an
3669 * aggregation over a single NIC that has no Tx rings. Even in such a
3670 * case, the single pseudo Tx ring will have a soft ring associated with
3671 * it and the soft ring will hang off the SRS.
3672 *
3673 * If a bandwidth is specified for an aggr, SRS_TX_BW_AGGR mode is used.
3674 * In this mode, the bandwidth is first applied on the outgoing packets
3675 * and later mac_tx_addr_mode() function is called to send the packet out
3676 * of one of the pseudo Tx rings.
3677 *
3678 * Four flags are used in srs_state for indicating flow control
3679 * conditions : SRS_TX_BLOCKED, SRS_TX_HIWAT, SRS_TX_WAKEUP_CLIENT.
3680 * SRS_TX_BLOCKED indicates out of Tx descs. SRS expects a wakeup from the
3681 * driver below.
3682 * SRS_TX_HIWAT indicates packet count enqueued in Tx SRS exceeded Tx hiwat
3683 * and flow-control pressure is applied back to clients. The clients expect
3684 * wakeup when flow-control is relieved.
3685 * SRS_TX_WAKEUP_CLIENT get set when (flag == MAC_TX_NO_ENQUEUE) and mblk
3686 * got returned back to client either due to lack of Tx descs or due to bw
3687 * control reasons. The clients expect a wakeup when condition is relieved.
3688 *
3689 * The fourth argument to mac_tx() is the flag. Normally it will be 0 but
3690 * some clients set the following values too: MAC_DROP_ON_NO_DESC,
3691 * MAC_TX_NO_ENQUEUE
3692 * Mac clients that do not want packets to be enqueued in the mac layer set
3693 * MAC_DROP_ON_NO_DESC value. The packets won't be queued in the Tx SRS or
3694 * Tx soft rings but instead get dropped when the NIC runs out of desc. The
3695 * behaviour of this flag is different when the Tx is running in serializer
3696 * or bandwidth mode. Under these (Serializer, bandwidth) modes, the packet
3697 * get dropped when Tx high watermark is reached.
3698 * There are some mac clients like vsw, aggr that want the mblks to be
3699 * returned back to clients instead of being queued in Tx SRS (or Tx soft
3700 * rings) under flow-control (i.e., out of desc or exceeding bw limits)
3701 * conditions. These clients call mac_tx() with MAC_TX_NO_ENQUEUE flag set.
3702 * In the default and Tx fanout mode, the un-transmitted mblks will be
3703 * returned back to the clients when the driver runs out of Tx descs.
3704 * SRS_TX_WAKEUP_CLIENT (or S_RING_WAKEUP_CLIENT) will be set in SRS (or
3705 * soft ring) so that the clients can be woken up when Tx desc become
3706 * available. When running in serializer or bandwidth mode mode,
3707 * SRS_TX_WAKEUP_CLIENT will be set when tx hi-watermark is reached.
3708 */
3709
3710 mac_tx_func_t
mac_tx_get_func(uint32_t mode)3711 mac_tx_get_func(uint32_t mode)
3712 {
3713 return (mac_tx_mode_list[mode].mac_tx_func);
3714 }
3715
3716 /* ARGSUSED */
3717 static mac_tx_cookie_t
mac_tx_single_ring_mode(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uintptr_t fanout_hint,uint16_t flag,mblk_t ** ret_mp)3718 mac_tx_single_ring_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3719 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3720 {
3721 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3722 mac_tx_stats_t stats;
3723 mac_tx_cookie_t cookie = 0;
3724
3725 ASSERT(srs_tx->st_mode == SRS_TX_DEFAULT);
3726
3727 /* Regular case with a single Tx ring */
3728 /*
3729 * SRS_TX_BLOCKED is set when underlying NIC runs
3730 * out of Tx descs and messages start getting
3731 * queued. It won't get reset until
3732 * tx_srs_drain() completely drains out the
3733 * messages.
3734 */
3735 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) {
3736 /* Tx descs/resources not available */
3737 mutex_enter(&mac_srs->srs_lock);
3738 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) {
3739 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain,
3740 flag, ret_mp);
3741 mutex_exit(&mac_srs->srs_lock);
3742 return (cookie);
3743 }
3744 /*
3745 * While we were computing mblk count, the
3746 * flow control condition got relieved.
3747 * Continue with the transmission.
3748 */
3749 mutex_exit(&mac_srs->srs_lock);
3750 }
3751
3752 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
3753 mp_chain, &stats);
3754
3755 /*
3756 * Multiple threads could be here sending packets.
3757 * Under such conditions, it is not possible to
3758 * automically set SRS_TX_BLOCKED bit to indicate
3759 * out of tx desc condition. To atomically set
3760 * this, we queue the returned packet and do
3761 * the setting of SRS_TX_BLOCKED in
3762 * mac_tx_srs_drain().
3763 */
3764 if (mp_chain != NULL) {
3765 mutex_enter(&mac_srs->srs_lock);
3766 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, flag, ret_mp);
3767 mutex_exit(&mac_srs->srs_lock);
3768 return (cookie);
3769 }
3770 SRS_TX_STATS_UPDATE(mac_srs, &stats);
3771
3772 return (0);
3773 }
3774
3775 /*
3776 * mac_tx_serialize_mode
3777 *
3778 * This is an experimental mode implemented as per the request of PAE.
3779 * In this mode, all callers attempting to send a packet to the NIC
3780 * will get serialized. Only one thread at any time will access the
3781 * NIC to send the packet out.
3782 */
3783 /* ARGSUSED */
3784 static mac_tx_cookie_t
mac_tx_serializer_mode(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uintptr_t fanout_hint,uint16_t flag,mblk_t ** ret_mp)3785 mac_tx_serializer_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3786 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3787 {
3788 mac_tx_stats_t stats;
3789 mac_tx_cookie_t cookie = 0;
3790 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3791
3792 /* Single ring, serialize below */
3793 ASSERT(srs_tx->st_mode == SRS_TX_SERIALIZE);
3794 mutex_enter(&mac_srs->srs_lock);
3795 if ((mac_srs->srs_first != NULL) ||
3796 (mac_srs->srs_state & SRS_PROC)) {
3797 /*
3798 * In serialization mode, queue all packets until
3799 * TX_HIWAT is set.
3800 * If drop bit is set, drop if TX_HIWAT is set.
3801 * If no_enqueue is set, still enqueue until hiwat
3802 * is set and return mblks after TX_HIWAT is set.
3803 */
3804 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain,
3805 flag, 0, ret_mp);
3806 mutex_exit(&mac_srs->srs_lock);
3807 return (cookie);
3808 }
3809 /*
3810 * No packets queued, nothing on proc and no flow
3811 * control condition. Fast-path, ok. Do inline
3812 * processing.
3813 */
3814 mac_srs->srs_state |= SRS_PROC;
3815 mutex_exit(&mac_srs->srs_lock);
3816
3817 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
3818 mp_chain, &stats);
3819
3820 mutex_enter(&mac_srs->srs_lock);
3821 mac_srs->srs_state &= ~SRS_PROC;
3822 if (mp_chain != NULL) {
3823 cookie = mac_tx_srs_enqueue(mac_srs,
3824 mp_chain, flag, 0, ret_mp);
3825 }
3826 if (mac_srs->srs_first != NULL) {
3827 /*
3828 * We processed inline our packet and a new
3829 * packet/s got queued while we were
3830 * processing. Wakeup srs worker
3831 */
3832 cv_signal(&mac_srs->srs_async);
3833 }
3834 mutex_exit(&mac_srs->srs_lock);
3835
3836 if (cookie == 0)
3837 SRS_TX_STATS_UPDATE(mac_srs, &stats);
3838
3839 return (cookie);
3840 }
3841
3842 /*
3843 * mac_tx_fanout_mode
3844 *
3845 * In this mode, the SRS will have access to multiple Tx rings to send
3846 * the packet out. The fanout hint that is passed as an argument is
3847 * used to find an appropriate ring to fanout the traffic. Each Tx
3848 * ring, in turn, will have a soft ring associated with it. If a Tx
3849 * ring runs out of Tx desc's the returned packet will be queued in
3850 * the soft ring associated with that Tx ring. The srs itself will not
3851 * queue any packets.
3852 */
3853
3854 #define MAC_TX_SOFT_RING_PROCESS(chain) { \
3855 index = COMPUTE_INDEX(hash, mac_srs->srs_tx_ring_count), \
3856 softring = mac_srs->srs_tx_soft_rings[index]; \
3857 cookie = mac_tx_soft_ring_process(softring, chain, flag, ret_mp); \
3858 DTRACE_PROBE2(tx__fanout, uint64_t, hash, uint_t, index); \
3859 }
3860
3861 static mac_tx_cookie_t
mac_tx_fanout_mode(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uintptr_t fanout_hint,uint16_t flag,mblk_t ** ret_mp)3862 mac_tx_fanout_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3863 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3864 {
3865 mac_soft_ring_t *softring;
3866 uint64_t hash;
3867 uint_t index;
3868 mac_tx_cookie_t cookie = 0;
3869
3870 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT ||
3871 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT);
3872 if (fanout_hint != 0) {
3873 /*
3874 * The hint is specified by the caller, simply pass the
3875 * whole chain to the soft ring.
3876 */
3877 hash = HASH_HINT(fanout_hint);
3878 MAC_TX_SOFT_RING_PROCESS(mp_chain);
3879 } else {
3880 mblk_t *last_mp, *cur_mp, *sub_chain;
3881 uint64_t last_hash = 0;
3882 uint_t media = mac_srs->srs_mcip->mci_mip->mi_info.mi_media;
3883
3884 /*
3885 * Compute the hash from the contents (headers) of the
3886 * packets of the mblk chain. Split the chains into
3887 * subchains of the same conversation.
3888 *
3889 * Since there may be more than one ring used for
3890 * sub-chains of the same call, and since the caller
3891 * does not maintain per conversation state since it
3892 * passed a zero hint, unsent subchains will be
3893 * dropped.
3894 */
3895
3896 flag |= MAC_DROP_ON_NO_DESC;
3897 ret_mp = NULL;
3898
3899 ASSERT(ret_mp == NULL);
3900
3901 sub_chain = NULL;
3902 last_mp = NULL;
3903
3904 for (cur_mp = mp_chain; cur_mp != NULL;
3905 cur_mp = cur_mp->b_next) {
3906 hash = mac_pkt_hash(media, cur_mp, MAC_PKT_HASH_L4,
3907 B_TRUE);
3908 if (last_hash != 0 && hash != last_hash) {
3909 /*
3910 * Starting a different subchain, send current
3911 * chain out.
3912 */
3913 ASSERT(last_mp != NULL);
3914 last_mp->b_next = NULL;
3915 MAC_TX_SOFT_RING_PROCESS(sub_chain);
3916 sub_chain = NULL;
3917 }
3918
3919 /* add packet to subchain */
3920 if (sub_chain == NULL)
3921 sub_chain = cur_mp;
3922 last_mp = cur_mp;
3923 last_hash = hash;
3924 }
3925
3926 if (sub_chain != NULL) {
3927 /* send last subchain */
3928 ASSERT(last_mp != NULL);
3929 last_mp->b_next = NULL;
3930 MAC_TX_SOFT_RING_PROCESS(sub_chain);
3931 }
3932
3933 cookie = 0;
3934 }
3935
3936 return (cookie);
3937 }
3938
3939 /*
3940 * mac_tx_bw_mode
3941 *
3942 * In the bandwidth mode, Tx srs will allow packets to go down to Tx ring
3943 * only if bw is available. Otherwise the packets will be queued in
3944 * SRS. If the SRS has multiple Tx rings, then packets will get fanned
3945 * out to a Tx rings.
3946 */
3947 static mac_tx_cookie_t
mac_tx_bw_mode(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uintptr_t fanout_hint,uint16_t flag,mblk_t ** ret_mp)3948 mac_tx_bw_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3949 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3950 {
3951 int cnt, sz;
3952 mblk_t *tail;
3953 mac_tx_cookie_t cookie = 0;
3954 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3955 clock_t now;
3956
3957 ASSERT(TX_BANDWIDTH_MODE(mac_srs));
3958 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL);
3959 mutex_enter(&mac_srs->srs_lock);
3960 if (mac_srs->srs_bw->mac_bw_limit == 0) {
3961 /*
3962 * zero bandwidth, no traffic is sent: drop the packets,
3963 * or return the whole chain if the caller requests all
3964 * unsent packets back.
3965 */
3966 if (flag & MAC_TX_NO_ENQUEUE) {
3967 cookie = (mac_tx_cookie_t)mac_srs;
3968 *ret_mp = mp_chain;
3969 } else {
3970 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie,
3971 "Tx no bandwidth");
3972 }
3973 mutex_exit(&mac_srs->srs_lock);
3974 return (cookie);
3975 } else if ((mac_srs->srs_first != NULL) ||
3976 (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
3977 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag,
3978 fanout_hint, ret_mp);
3979 mutex_exit(&mac_srs->srs_lock);
3980 return (cookie);
3981 }
3982 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3983 now = ddi_get_lbolt();
3984 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
3985 mac_srs->srs_bw->mac_bw_curr_time = now;
3986 mac_srs->srs_bw->mac_bw_used = 0;
3987 } else if (mac_srs->srs_bw->mac_bw_used >
3988 mac_srs->srs_bw->mac_bw_limit) {
3989 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
3990 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3991 mp_chain, tail, cnt, sz);
3992 /*
3993 * Wakeup worker thread. Note that worker
3994 * thread has to be woken up so that it
3995 * can fire up the timer to be woken up
3996 * on the next tick. Also once
3997 * BW_ENFORCED is set, it can only be
3998 * reset by srs_worker thread. Until then
3999 * all packets will get queued up in SRS
4000 * and hence this this code path won't be
4001 * entered until BW_ENFORCED is reset.
4002 */
4003 cv_signal(&mac_srs->srs_async);
4004 mutex_exit(&mac_srs->srs_lock);
4005 return (cookie);
4006 }
4007
4008 mac_srs->srs_bw->mac_bw_used += sz;
4009 mutex_exit(&mac_srs->srs_lock);
4010
4011 if (srs_tx->st_mode == SRS_TX_BW_FANOUT) {
4012 mac_soft_ring_t *softring;
4013 uint_t indx, hash;
4014
4015 hash = HASH_HINT(fanout_hint);
4016 indx = COMPUTE_INDEX(hash,
4017 mac_srs->srs_tx_ring_count);
4018 softring = mac_srs->srs_tx_soft_rings[indx];
4019 return (mac_tx_soft_ring_process(softring, mp_chain, flag,
4020 ret_mp));
4021 } else if (srs_tx->st_mode == SRS_TX_BW_AGGR) {
4022 return (mac_tx_aggr_mode(mac_srs, mp_chain,
4023 fanout_hint, flag, ret_mp));
4024 } else {
4025 mac_tx_stats_t stats;
4026
4027 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
4028 mp_chain, &stats);
4029
4030 if (mp_chain != NULL) {
4031 mutex_enter(&mac_srs->srs_lock);
4032 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
4033 if (mac_srs->srs_bw->mac_bw_used > sz)
4034 mac_srs->srs_bw->mac_bw_used -= sz;
4035 else
4036 mac_srs->srs_bw->mac_bw_used = 0;
4037 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag,
4038 fanout_hint, ret_mp);
4039 mutex_exit(&mac_srs->srs_lock);
4040 return (cookie);
4041 }
4042 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4043
4044 return (0);
4045 }
4046 }
4047
4048 /*
4049 * mac_tx_aggr_mode
4050 *
4051 * This routine invokes an aggr function, aggr_find_tx_ring(), to find
4052 * a (pseudo) Tx ring belonging to a port on which the packet has to
4053 * be sent. aggr_find_tx_ring() first finds the outgoing port based on
4054 * L2/L3/L4 policy and then uses the fanout_hint passed to it to pick
4055 * a Tx ring from the selected port.
4056 *
4057 * Note that a port can be deleted from the aggregation. In such a case,
4058 * the aggregation layer first separates the port from the rest of the
4059 * ports making sure that port (and thus any Tx rings associated with
4060 * it) won't get selected in the call to aggr_find_tx_ring() function.
4061 * Later calls are made to mac_group_rem_ring() passing pseudo Tx ring
4062 * handles one by one which in turn will quiesce the Tx SRS and remove
4063 * the soft ring associated with the pseudo Tx ring. Unlike Rx side
4064 * where a cookie is used to protect against mac_rx_ring() calls on
4065 * rings that have been removed, no such cookie is needed on the Tx
4066 * side as the pseudo Tx ring won't be available anymore to
4067 * aggr_find_tx_ring() once the port has been removed.
4068 */
4069 static mac_tx_cookie_t
mac_tx_aggr_mode(mac_soft_ring_set_t * mac_srs,mblk_t * mp_chain,uintptr_t fanout_hint,uint16_t flag,mblk_t ** ret_mp)4070 mac_tx_aggr_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
4071 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
4072 {
4073 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
4074 mac_tx_ring_fn_t find_tx_ring_fn;
4075 mac_ring_handle_t ring = NULL;
4076 void *arg;
4077 mac_soft_ring_t *sringp;
4078
4079 find_tx_ring_fn = srs_tx->st_capab_aggr.mca_find_tx_ring_fn;
4080 arg = srs_tx->st_capab_aggr.mca_arg;
4081 if (find_tx_ring_fn(arg, mp_chain, fanout_hint, &ring) == NULL)
4082 return (0);
4083 sringp = srs_tx->st_soft_rings[((mac_ring_t *)ring)->mr_index];
4084 return (mac_tx_soft_ring_process(sringp, mp_chain, flag, ret_mp));
4085 }
4086
4087 void
mac_tx_invoke_callbacks(mac_client_impl_t * mcip,mac_tx_cookie_t cookie)4088 mac_tx_invoke_callbacks(mac_client_impl_t *mcip, mac_tx_cookie_t cookie)
4089 {
4090 mac_cb_t *mcb;
4091 mac_tx_notify_cb_t *mtnfp;
4092
4093 /* Wakeup callback registered clients */
4094 MAC_CALLBACK_WALKER_INC(&mcip->mci_tx_notify_cb_info);
4095 for (mcb = mcip->mci_tx_notify_cb_list; mcb != NULL;
4096 mcb = mcb->mcb_nextp) {
4097 mtnfp = (mac_tx_notify_cb_t *)mcb->mcb_objp;
4098 mtnfp->mtnf_fn(mtnfp->mtnf_arg, cookie);
4099 }
4100 MAC_CALLBACK_WALKER_DCR(&mcip->mci_tx_notify_cb_info,
4101 &mcip->mci_tx_notify_cb_list);
4102 }
4103
4104 /* ARGSUSED */
4105 void
mac_tx_srs_drain(mac_soft_ring_set_t * mac_srs,uint_t proc_type)4106 mac_tx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
4107 {
4108 mblk_t *head, *tail;
4109 size_t sz;
4110 uint32_t tx_mode;
4111 uint_t saved_pkt_count;
4112 mac_tx_stats_t stats;
4113 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
4114 clock_t now;
4115
4116 saved_pkt_count = 0;
4117 ASSERT(mutex_owned(&mac_srs->srs_lock));
4118 ASSERT(!(mac_srs->srs_state & SRS_PROC));
4119
4120 mac_srs->srs_state |= SRS_PROC;
4121
4122 tx_mode = srs_tx->st_mode;
4123 if (tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_SERIALIZE) {
4124 if (mac_srs->srs_first != NULL) {
4125 head = mac_srs->srs_first;
4126 tail = mac_srs->srs_last;
4127 saved_pkt_count = mac_srs->srs_count;
4128 mac_srs->srs_first = NULL;
4129 mac_srs->srs_last = NULL;
4130 mac_srs->srs_count = 0;
4131 mutex_exit(&mac_srs->srs_lock);
4132
4133 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
4134 head, &stats);
4135
4136 mutex_enter(&mac_srs->srs_lock);
4137 if (head != NULL) {
4138 /* Device out of tx desc, set block */
4139 if (head->b_next == NULL)
4140 VERIFY(head == tail);
4141 tail->b_next = mac_srs->srs_first;
4142 mac_srs->srs_first = head;
4143 mac_srs->srs_count +=
4144 (saved_pkt_count - stats.mts_opackets);
4145 if (mac_srs->srs_last == NULL)
4146 mac_srs->srs_last = tail;
4147 MAC_TX_SRS_BLOCK(mac_srs, head);
4148 } else {
4149 srs_tx->st_woken_up = B_FALSE;
4150 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4151 }
4152 }
4153 } else if (tx_mode == SRS_TX_BW) {
4154 /*
4155 * We are here because the timer fired and we have some data
4156 * to tranmit. Also mac_tx_srs_worker should have reset
4157 * SRS_BW_ENFORCED flag
4158 */
4159 ASSERT(!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED));
4160 head = tail = mac_srs->srs_first;
4161 while (mac_srs->srs_first != NULL) {
4162 tail = mac_srs->srs_first;
4163 tail->b_prev = NULL;
4164 mac_srs->srs_first = tail->b_next;
4165 if (mac_srs->srs_first == NULL)
4166 mac_srs->srs_last = NULL;
4167 mac_srs->srs_count--;
4168 sz = msgdsize(tail);
4169 mac_srs->srs_size -= sz;
4170 saved_pkt_count++;
4171 MAC_TX_UPDATE_BW_INFO(mac_srs, sz);
4172
4173 if (mac_srs->srs_bw->mac_bw_used <
4174 mac_srs->srs_bw->mac_bw_limit)
4175 continue;
4176
4177 now = ddi_get_lbolt();
4178 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
4179 mac_srs->srs_bw->mac_bw_curr_time = now;
4180 mac_srs->srs_bw->mac_bw_used = sz;
4181 continue;
4182 }
4183 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
4184 break;
4185 }
4186
4187 ASSERT((head == NULL && tail == NULL) ||
4188 (head != NULL && tail != NULL));
4189 if (tail != NULL) {
4190 tail->b_next = NULL;
4191 mutex_exit(&mac_srs->srs_lock);
4192
4193 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
4194 head, &stats);
4195
4196 mutex_enter(&mac_srs->srs_lock);
4197 if (head != NULL) {
4198 uint_t size_sent;
4199
4200 /* Device out of tx desc, set block */
4201 if (head->b_next == NULL)
4202 VERIFY(head == tail);
4203 tail->b_next = mac_srs->srs_first;
4204 mac_srs->srs_first = head;
4205 mac_srs->srs_count +=
4206 (saved_pkt_count - stats.mts_opackets);
4207 if (mac_srs->srs_last == NULL)
4208 mac_srs->srs_last = tail;
4209 size_sent = sz - stats.mts_obytes;
4210 mac_srs->srs_size += size_sent;
4211 mac_srs->srs_bw->mac_bw_sz += size_sent;
4212 if (mac_srs->srs_bw->mac_bw_used > size_sent) {
4213 mac_srs->srs_bw->mac_bw_used -=
4214 size_sent;
4215 } else {
4216 mac_srs->srs_bw->mac_bw_used = 0;
4217 }
4218 MAC_TX_SRS_BLOCK(mac_srs, head);
4219 } else {
4220 srs_tx->st_woken_up = B_FALSE;
4221 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4222 }
4223 }
4224 } else if (tx_mode == SRS_TX_BW_FANOUT || tx_mode == SRS_TX_BW_AGGR) {
4225 mblk_t *prev;
4226 uint64_t hint;
4227
4228 /*
4229 * We are here because the timer fired and we
4230 * have some quota to tranmit.
4231 */
4232 prev = NULL;
4233 head = tail = mac_srs->srs_first;
4234 while (mac_srs->srs_first != NULL) {
4235 tail = mac_srs->srs_first;
4236 mac_srs->srs_first = tail->b_next;
4237 if (mac_srs->srs_first == NULL)
4238 mac_srs->srs_last = NULL;
4239 mac_srs->srs_count--;
4240 sz = msgdsize(tail);
4241 mac_srs->srs_size -= sz;
4242 mac_srs->srs_bw->mac_bw_used += sz;
4243 if (prev == NULL)
4244 hint = (ulong_t)tail->b_prev;
4245 if (hint != (ulong_t)tail->b_prev) {
4246 prev->b_next = NULL;
4247 mutex_exit(&mac_srs->srs_lock);
4248 TX_SRS_TO_SOFT_RING(mac_srs, head, hint);
4249 head = tail;
4250 hint = (ulong_t)tail->b_prev;
4251 mutex_enter(&mac_srs->srs_lock);
4252 }
4253
4254 prev = tail;
4255 tail->b_prev = NULL;
4256 if (mac_srs->srs_bw->mac_bw_used <
4257 mac_srs->srs_bw->mac_bw_limit)
4258 continue;
4259
4260 now = ddi_get_lbolt();
4261 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
4262 mac_srs->srs_bw->mac_bw_curr_time = now;
4263 mac_srs->srs_bw->mac_bw_used = 0;
4264 continue;
4265 }
4266 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
4267 break;
4268 }
4269 ASSERT((head == NULL && tail == NULL) ||
4270 (head != NULL && tail != NULL));
4271 if (tail != NULL) {
4272 tail->b_next = NULL;
4273 mutex_exit(&mac_srs->srs_lock);
4274 TX_SRS_TO_SOFT_RING(mac_srs, head, hint);
4275 mutex_enter(&mac_srs->srs_lock);
4276 }
4277 }
4278 /*
4279 * SRS_TX_FANOUT case not considered here because packets
4280 * won't be queued in the SRS for this case. Packets will
4281 * be sent directly to soft rings underneath and if there
4282 * is any queueing at all, it would be in Tx side soft
4283 * rings.
4284 */
4285
4286 /*
4287 * When srs_count becomes 0, reset SRS_TX_HIWAT and
4288 * SRS_TX_WAKEUP_CLIENT and wakeup registered clients.
4289 */
4290 if (mac_srs->srs_count == 0 && (mac_srs->srs_state &
4291 (SRS_TX_HIWAT | SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED))) {
4292 mac_client_impl_t *mcip = mac_srs->srs_mcip;
4293 boolean_t wakeup_required = B_FALSE;
4294
4295 if (mac_srs->srs_state &
4296 (SRS_TX_HIWAT|SRS_TX_WAKEUP_CLIENT)) {
4297 wakeup_required = B_TRUE;
4298 }
4299 mac_srs->srs_state &= ~(SRS_TX_HIWAT |
4300 SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED);
4301 mutex_exit(&mac_srs->srs_lock);
4302 if (wakeup_required) {
4303 mac_tx_invoke_callbacks(mcip, (mac_tx_cookie_t)mac_srs);
4304 /*
4305 * If the client is not the primary MAC client, then we
4306 * need to send the notification to the clients upper
4307 * MAC, i.e. mci_upper_mip.
4308 */
4309 mac_tx_notify(mcip->mci_upper_mip != NULL ?
4310 mcip->mci_upper_mip : mcip->mci_mip);
4311 }
4312 mutex_enter(&mac_srs->srs_lock);
4313 }
4314 mac_srs->srs_state &= ~SRS_PROC;
4315 }
4316
4317 /*
4318 * Given a packet, get the flow_entry that identifies the flow
4319 * to which that packet belongs. The flow_entry will contain
4320 * the transmit function to be used to send the packet. If the
4321 * function returns NULL, the packet should be sent using the
4322 * underlying NIC.
4323 */
4324 static flow_entry_t *
mac_tx_classify(mac_impl_t * mip,mblk_t * mp)4325 mac_tx_classify(mac_impl_t *mip, mblk_t *mp)
4326 {
4327 flow_entry_t *flent = NULL;
4328 mac_client_impl_t *mcip;
4329 int err;
4330
4331 /*
4332 * Do classification on the packet.
4333 */
4334 err = mac_flow_lookup(mip->mi_flow_tab, mp, FLOW_OUTBOUND, &flent);
4335 if (err != 0)
4336 return (NULL);
4337
4338 /*
4339 * This flent might just be an additional one on the MAC client,
4340 * i.e. for classification purposes (different fdesc), however
4341 * the resources, SRS et. al., are in the mci_flent, so if
4342 * this isn't the mci_flent, we need to get it.
4343 */
4344 if ((mcip = flent->fe_mcip) != NULL && mcip->mci_flent != flent) {
4345 FLOW_REFRELE(flent);
4346 flent = mcip->mci_flent;
4347 FLOW_TRY_REFHOLD(flent, err);
4348 if (err != 0)
4349 return (NULL);
4350 }
4351
4352 return (flent);
4353 }
4354
4355 /*
4356 * This macro is only meant to be used by mac_tx_send().
4357 */
4358 #define CHECK_VID_AND_ADD_TAG(mp) { \
4359 if (vid_check) { \
4360 int err = 0; \
4361 \
4362 MAC_VID_CHECK(src_mcip, (mp), err); \
4363 if (err != 0) { \
4364 freemsg((mp)); \
4365 (mp) = next; \
4366 oerrors++; \
4367 continue; \
4368 } \
4369 } \
4370 if (add_tag) { \
4371 (mp) = mac_add_vlan_tag((mp), 0, vid); \
4372 if ((mp) == NULL) { \
4373 (mp) = next; \
4374 oerrors++; \
4375 continue; \
4376 } \
4377 } \
4378 }
4379
4380 mblk_t *
mac_tx_send(mac_client_handle_t mch,mac_ring_handle_t ring,mblk_t * mp_chain,mac_tx_stats_t * stats)4381 mac_tx_send(mac_client_handle_t mch, mac_ring_handle_t ring, mblk_t *mp_chain,
4382 mac_tx_stats_t *stats)
4383 {
4384 mac_client_impl_t *src_mcip = (mac_client_impl_t *)mch;
4385 mac_impl_t *mip = src_mcip->mci_mip;
4386 uint_t obytes = 0, opackets = 0, oerrors = 0;
4387 mblk_t *mp = NULL, *next;
4388 boolean_t vid_check, add_tag;
4389 uint16_t vid = 0;
4390
4391 if (mip->mi_nclients > 1) {
4392 vid_check = MAC_VID_CHECK_NEEDED(src_mcip);
4393 add_tag = MAC_TAG_NEEDED(src_mcip);
4394 if (add_tag)
4395 vid = mac_client_vid(mch);
4396 } else {
4397 ASSERT(mip->mi_nclients == 1);
4398 vid_check = add_tag = B_FALSE;
4399 }
4400
4401 /*
4402 * Fastpath: if there's only one client, we simply send
4403 * the packet down to the underlying NIC.
4404 */
4405 if (mip->mi_nactiveclients == 1) {
4406 DTRACE_PROBE2(fastpath,
4407 mac_client_impl_t *, src_mcip, mblk_t *, mp_chain);
4408
4409 mp = mp_chain;
4410 while (mp != NULL) {
4411 next = mp->b_next;
4412 mp->b_next = NULL;
4413 opackets++;
4414 obytes += (mp->b_cont == NULL ? MBLKL(mp) :
4415 msgdsize(mp));
4416
4417 CHECK_VID_AND_ADD_TAG(mp);
4418 mp = mac_provider_tx(mip, ring, mp, src_mcip);
4419
4420 /*
4421 * If the driver is out of descriptors and does a
4422 * partial send it will return a chain of unsent
4423 * mblks. Adjust the accounting stats.
4424 */
4425 if (mp != NULL) {
4426 opackets--;
4427 obytes -= msgdsize(mp);
4428 mp->b_next = next;
4429 break;
4430 }
4431 mp = next;
4432 }
4433 goto done;
4434 }
4435
4436 /*
4437 * No fastpath, we either have more than one MAC client
4438 * defined on top of the same MAC, or one or more MAC
4439 * client promiscuous callbacks.
4440 */
4441 DTRACE_PROBE3(slowpath, mac_client_impl_t *,
4442 src_mcip, int, mip->mi_nclients, mblk_t *, mp_chain);
4443
4444 mp = mp_chain;
4445 while (mp != NULL) {
4446 flow_entry_t *dst_flow_ent;
4447 void *flow_cookie;
4448 size_t pkt_size;
4449
4450 next = mp->b_next;
4451 mp->b_next = NULL;
4452 opackets++;
4453 pkt_size = (mp->b_cont == NULL ? MBLKL(mp) : msgdsize(mp));
4454 obytes += pkt_size;
4455 CHECK_VID_AND_ADD_TAG(mp);
4456
4457 /*
4458 * Find the destination.
4459 */
4460 dst_flow_ent = mac_tx_classify(mip, mp);
4461
4462 if (dst_flow_ent != NULL) {
4463 /*
4464 * Got a matching flow. It's either another
4465 * MAC client, or a broadcast/multicast flow.
4466 */
4467 flow_cookie = mac_flow_get_client_cookie(dst_flow_ent);
4468
4469 if (flow_cookie != NULL) {
4470 /*
4471 * The vnic_bcast_send function expects
4472 * to receive the sender MAC client
4473 * as value for arg2.
4474 */
4475 mac_bcast_send(flow_cookie, src_mcip, mp,
4476 B_TRUE);
4477 } else {
4478 /*
4479 * loopback the packet to a local MAC
4480 * client. We force a context switch
4481 * if both source and destination MAC
4482 * clients are used by IP, i.e.
4483 * bypass is set.
4484 */
4485 boolean_t do_switch;
4486
4487 mac_client_impl_t *dst_mcip =
4488 dst_flow_ent->fe_mcip;
4489
4490 /*
4491 * Check if there are promiscuous mode
4492 * callbacks defined. This check is
4493 * done here in the 'else' case and
4494 * not in other cases because this
4495 * path is for local loopback
4496 * communication which does not go
4497 * through MAC_TX(). For paths that go
4498 * through MAC_TX(), the promisc_list
4499 * check is done inside the MAC_TX()
4500 * macro.
4501 */
4502 if (mip->mi_promisc_list != NULL) {
4503 mac_promisc_dispatch(mip, mp, src_mcip,
4504 B_TRUE);
4505 }
4506
4507 do_switch = ((src_mcip->mci_state_flags &
4508 dst_mcip->mci_state_flags &
4509 MCIS_CLIENT_POLL_CAPABLE) != 0);
4510
4511 mac_hw_emul(&mp, NULL, NULL, MAC_ALL_EMULS);
4512 if (mp != NULL) {
4513 (dst_flow_ent->fe_cb_fn)(
4514 dst_flow_ent->fe_cb_arg1,
4515 dst_flow_ent->fe_cb_arg2,
4516 mp, do_switch);
4517 }
4518
4519 }
4520 FLOW_REFRELE(dst_flow_ent);
4521 } else {
4522 /*
4523 * Unknown destination, send via the underlying
4524 * NIC.
4525 */
4526 mp = mac_provider_tx(mip, ring, mp, src_mcip);
4527 if (mp != NULL) {
4528 /*
4529 * Adjust for the last packet that
4530 * could not be transmitted
4531 */
4532 opackets--;
4533 obytes -= pkt_size;
4534 mp->b_next = next;
4535 break;
4536 }
4537 }
4538 mp = next;
4539 }
4540
4541 done:
4542 stats->mts_obytes = obytes;
4543 stats->mts_opackets = opackets;
4544 stats->mts_oerrors = oerrors;
4545 return (mp);
4546 }
4547
4548 /*
4549 * mac_tx_srs_ring_present
4550 *
4551 * Returns whether the specified ring is part of the specified SRS.
4552 */
4553 boolean_t
mac_tx_srs_ring_present(mac_soft_ring_set_t * srs,mac_ring_t * tx_ring)4554 mac_tx_srs_ring_present(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring)
4555 {
4556 int i;
4557 mac_soft_ring_t *soft_ring;
4558
4559 if (srs->srs_tx.st_arg2 == tx_ring)
4560 return (B_TRUE);
4561
4562 for (i = 0; i < srs->srs_tx_ring_count; i++) {
4563 soft_ring = srs->srs_tx_soft_rings[i];
4564 if (soft_ring->s_ring_tx_arg2 == tx_ring)
4565 return (B_TRUE);
4566 }
4567
4568 return (B_FALSE);
4569 }
4570
4571 /*
4572 * mac_tx_srs_get_soft_ring
4573 *
4574 * Returns the TX soft ring associated with the given ring, if present.
4575 */
4576 mac_soft_ring_t *
mac_tx_srs_get_soft_ring(mac_soft_ring_set_t * srs,mac_ring_t * tx_ring)4577 mac_tx_srs_get_soft_ring(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring)
4578 {
4579 int i;
4580 mac_soft_ring_t *soft_ring;
4581
4582 if (srs->srs_tx.st_arg2 == tx_ring)
4583 return (NULL);
4584
4585 for (i = 0; i < srs->srs_tx_ring_count; i++) {
4586 soft_ring = srs->srs_tx_soft_rings[i];
4587 if (soft_ring->s_ring_tx_arg2 == tx_ring)
4588 return (soft_ring);
4589 }
4590
4591 return (NULL);
4592 }
4593
4594 /*
4595 * mac_tx_srs_wakeup
4596 *
4597 * Called when Tx desc become available. Wakeup the appropriate worker
4598 * thread after resetting the SRS_TX_BLOCKED/S_RING_BLOCK bit in the
4599 * state field.
4600 */
4601 void
mac_tx_srs_wakeup(mac_soft_ring_set_t * mac_srs,mac_ring_handle_t ring)4602 mac_tx_srs_wakeup(mac_soft_ring_set_t *mac_srs, mac_ring_handle_t ring)
4603 {
4604 int i;
4605 mac_soft_ring_t *sringp;
4606 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
4607
4608 mutex_enter(&mac_srs->srs_lock);
4609 /*
4610 * srs_tx_ring_count == 0 is the single ring mode case. In
4611 * this mode, there will not be Tx soft rings associated
4612 * with the SRS.
4613 */
4614 if (!MAC_TX_SOFT_RINGS(mac_srs)) {
4615 if (srs_tx->st_arg2 == ring &&
4616 mac_srs->srs_state & SRS_TX_BLOCKED) {
4617 mac_srs->srs_state &= ~SRS_TX_BLOCKED;
4618 srs_tx->st_stat.mts_unblockcnt++;
4619 cv_signal(&mac_srs->srs_async);
4620 }
4621 /*
4622 * A wakeup can come before tx_srs_drain() could
4623 * grab srs lock and set SRS_TX_BLOCKED. So
4624 * always set woken_up flag when we come here.
4625 */
4626 srs_tx->st_woken_up = B_TRUE;
4627 mutex_exit(&mac_srs->srs_lock);
4628 return;
4629 }
4630
4631 /*
4632 * If you are here, it is for FANOUT, BW_FANOUT,
4633 * AGGR_MODE or AGGR_BW_MODE case
4634 */
4635 for (i = 0; i < mac_srs->srs_tx_ring_count; i++) {
4636 sringp = mac_srs->srs_tx_soft_rings[i];
4637 mutex_enter(&sringp->s_ring_lock);
4638 if (sringp->s_ring_tx_arg2 == ring) {
4639 if (sringp->s_ring_state & S_RING_BLOCK) {
4640 sringp->s_ring_state &= ~S_RING_BLOCK;
4641 sringp->s_st_stat.mts_unblockcnt++;
4642 cv_signal(&sringp->s_ring_async);
4643 }
4644 sringp->s_ring_tx_woken_up = B_TRUE;
4645 }
4646 mutex_exit(&sringp->s_ring_lock);
4647 }
4648 mutex_exit(&mac_srs->srs_lock);
4649 }
4650
4651 /*
4652 * Once the driver is done draining, send a MAC_NOTE_TX notification to unleash
4653 * the blocked clients again.
4654 */
4655 void
mac_tx_notify(mac_impl_t * mip)4656 mac_tx_notify(mac_impl_t *mip)
4657 {
4658 i_mac_notify(mip, MAC_NOTE_TX);
4659 }
4660
4661 /*
4662 * RX SOFTRING RELATED FUNCTIONS
4663 *
4664 * These functions really belong in mac_soft_ring.c and here for
4665 * a short period.
4666 */
4667
4668 #define SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \
4669 /* \
4670 * Enqueue our mblk chain. \
4671 */ \
4672 ASSERT(MUTEX_HELD(&(ringp)->s_ring_lock)); \
4673 \
4674 if ((ringp)->s_ring_last != NULL) \
4675 (ringp)->s_ring_last->b_next = (mp); \
4676 else \
4677 (ringp)->s_ring_first = (mp); \
4678 (ringp)->s_ring_last = (tail); \
4679 (ringp)->s_ring_count += (cnt); \
4680 ASSERT((ringp)->s_ring_count > 0); \
4681 if ((ringp)->s_ring_type & ST_RING_BW_CTL) { \
4682 (ringp)->s_ring_size += sz; \
4683 } \
4684 }
4685
4686 /*
4687 * Default entry point to deliver a packet chain to a MAC client.
4688 * If the MAC client has flows, do the classification with these
4689 * flows as well.
4690 */
4691 /* ARGSUSED */
4692 void
mac_rx_deliver(void * arg1,mac_resource_handle_t mrh,mblk_t * mp_chain,mac_header_info_t * arg3)4693 mac_rx_deliver(void *arg1, mac_resource_handle_t mrh, mblk_t *mp_chain,
4694 mac_header_info_t *arg3)
4695 {
4696 mac_client_impl_t *mcip = arg1;
4697
4698 if (mcip->mci_nvids == 1 &&
4699 !(mcip->mci_state_flags & MCIS_STRIP_DISABLE)) {
4700 /*
4701 * If the client has exactly one VID associated with it
4702 * and striping of VLAN header is not disabled,
4703 * remove the VLAN tag from the packet before
4704 * passing it on to the client's receive callback.
4705 * Note that this needs to be done after we dispatch
4706 * the packet to the promiscuous listeners of the
4707 * client, since they expect to see the whole
4708 * frame including the VLAN headers.
4709 *
4710 * The MCIS_STRIP_DISABLE is only issued when sun4v
4711 * vsw is in play.
4712 */
4713 mp_chain = mac_strip_vlan_tag_chain(mp_chain);
4714 }
4715
4716 mcip->mci_rx_fn(mcip->mci_rx_arg, mrh, mp_chain, B_FALSE);
4717 }
4718
4719 /*
4720 * Process a chain for a given soft ring. If the number of packets
4721 * queued in the SRS and its associated soft rings (including this
4722 * one) is very small (tracked by srs_poll_pkt_cnt) then allow the
4723 * entering thread (interrupt or poll thread) to process the chain
4724 * inline. This is meant to reduce latency under low load.
4725 *
4726 * The proc and arg for each mblk is already stored in the mblk in
4727 * appropriate places.
4728 */
4729 /* ARGSUSED */
4730 void
mac_rx_soft_ring_process(mac_client_impl_t * mcip,mac_soft_ring_t * ringp,mblk_t * mp_chain,mblk_t * tail,int cnt,size_t sz)4731 mac_rx_soft_ring_process(mac_client_impl_t *mcip, mac_soft_ring_t *ringp,
4732 mblk_t *mp_chain, mblk_t *tail, int cnt, size_t sz)
4733 {
4734 mac_direct_rx_t proc;
4735 void *arg1;
4736 mac_resource_handle_t arg2;
4737 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4738
4739 ASSERT(ringp != NULL);
4740 ASSERT(mp_chain != NULL);
4741 ASSERT(tail != NULL);
4742 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4743
4744 mutex_enter(&ringp->s_ring_lock);
4745 ringp->s_ring_total_inpkt += cnt;
4746 ringp->s_ring_total_rbytes += sz;
4747 if ((mac_srs->srs_rx.sr_poll_pkt_cnt <= 1) &&
4748 !(ringp->s_ring_type & ST_RING_WORKER_ONLY)) {
4749 /* If on processor or blanking on, then enqueue and return */
4750 if (ringp->s_ring_state & S_RING_BLANK ||
4751 ringp->s_ring_state & S_RING_PROC) {
4752 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4753 mutex_exit(&ringp->s_ring_lock);
4754 return;
4755 }
4756 proc = ringp->s_ring_rx_func;
4757 arg1 = ringp->s_ring_rx_arg1;
4758 arg2 = ringp->s_ring_rx_arg2;
4759 /*
4760 * See if anything is already queued. If we are the
4761 * first packet, do inline processing else queue the
4762 * packet and do the drain.
4763 */
4764 if (ringp->s_ring_first == NULL) {
4765 /*
4766 * Fast-path, ok to process and nothing queued.
4767 */
4768 ringp->s_ring_run = curthread;
4769 ringp->s_ring_state |= (S_RING_PROC);
4770
4771 mutex_exit(&ringp->s_ring_lock);
4772
4773 /*
4774 * We are the chain of 1 packet so
4775 * go through this fast path.
4776 */
4777 ASSERT(mp_chain->b_next == NULL);
4778
4779 (*proc)(arg1, arg2, mp_chain, NULL);
4780
4781 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4782 /*
4783 * If we have an SRS performing bandwidth
4784 * control then we need to decrement the size
4785 * and count so the SRS has an accurate count
4786 * of the data queued between the SRS and its
4787 * soft rings. We decrement the counters only
4788 * when the packet is processed by both the
4789 * SRS and the soft ring.
4790 */
4791 mutex_enter(&mac_srs->srs_lock);
4792 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
4793 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz);
4794 mutex_exit(&mac_srs->srs_lock);
4795
4796 mutex_enter(&ringp->s_ring_lock);
4797 ringp->s_ring_run = NULL;
4798 ringp->s_ring_state &= ~S_RING_PROC;
4799 if (ringp->s_ring_state & S_RING_CLIENT_WAIT)
4800 cv_signal(&ringp->s_ring_client_cv);
4801
4802 if ((ringp->s_ring_first == NULL) ||
4803 (ringp->s_ring_state & S_RING_BLANK)) {
4804 /*
4805 * We processed a single packet inline
4806 * and nothing new has arrived or our
4807 * receiver doesn't want to receive
4808 * any packets. We are done.
4809 */
4810 mutex_exit(&ringp->s_ring_lock);
4811 return;
4812 }
4813 } else {
4814 SOFT_RING_ENQUEUE_CHAIN(ringp,
4815 mp_chain, tail, cnt, sz);
4816 }
4817
4818 /*
4819 * We are here because either we couldn't do inline
4820 * processing (because something was already
4821 * queued), or we had a chain of more than one
4822 * packet, or something else arrived after we were
4823 * done with inline processing.
4824 */
4825 ASSERT(MUTEX_HELD(&ringp->s_ring_lock));
4826 ASSERT(ringp->s_ring_first != NULL);
4827
4828 ringp->s_ring_drain_func(ringp);
4829 mutex_exit(&ringp->s_ring_lock);
4830 return;
4831 } else {
4832 /* ST_RING_WORKER_ONLY case */
4833 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4834 mac_soft_ring_worker_wakeup(ringp);
4835 mutex_exit(&ringp->s_ring_lock);
4836 }
4837 }
4838
4839 /*
4840 * TX SOFTRING RELATED FUNCTIONS
4841 *
4842 * These functions really belong in mac_soft_ring.c and here for
4843 * a short period.
4844 */
4845
4846 #define TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \
4847 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); \
4848 ringp->s_ring_state |= S_RING_ENQUEUED; \
4849 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); \
4850 }
4851
4852 /*
4853 * mac_tx_sring_queued
4854 *
4855 * When we are out of transmit descriptors and we already have a
4856 * queue that exceeds hiwat (or the client called us with
4857 * MAC_TX_NO_ENQUEUE or MAC_DROP_ON_NO_DESC flag), return the
4858 * soft ring pointer as the opaque cookie for the client enable
4859 * flow control.
4860 */
4861 static mac_tx_cookie_t
mac_tx_sring_enqueue(mac_soft_ring_t * ringp,mblk_t * mp_chain,uint16_t flag,mblk_t ** ret_mp)4862 mac_tx_sring_enqueue(mac_soft_ring_t *ringp, mblk_t *mp_chain, uint16_t flag,
4863 mblk_t **ret_mp)
4864 {
4865 int cnt;
4866 size_t sz;
4867 mblk_t *tail;
4868 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4869 mac_tx_cookie_t cookie = 0;
4870 boolean_t wakeup_worker = B_TRUE;
4871
4872 ASSERT(MUTEX_HELD(&ringp->s_ring_lock));
4873 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
4874 if (flag & MAC_DROP_ON_NO_DESC) {
4875 mac_drop_chain(mp_chain, "Tx softring no desc");
4876 /* increment freed stats */
4877 ringp->s_ring_drops += cnt;
4878 cookie = (mac_tx_cookie_t)ringp;
4879 } else {
4880 if (ringp->s_ring_first != NULL)
4881 wakeup_worker = B_FALSE;
4882
4883 if (flag & MAC_TX_NO_ENQUEUE) {
4884 /*
4885 * If QUEUED is not set, queue the packet
4886 * and let mac_tx_soft_ring_drain() set
4887 * the TX_BLOCKED bit for the reasons
4888 * explained above. Otherwise, return the
4889 * mblks.
4890 */
4891 if (wakeup_worker) {
4892 TX_SOFT_RING_ENQUEUE_CHAIN(ringp,
4893 mp_chain, tail, cnt, sz);
4894 } else {
4895 ringp->s_ring_state |= S_RING_WAKEUP_CLIENT;
4896 cookie = (mac_tx_cookie_t)ringp;
4897 *ret_mp = mp_chain;
4898 }
4899 } else {
4900 boolean_t enqueue = B_TRUE;
4901
4902 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) {
4903 /*
4904 * flow-controlled. Store ringp in cookie
4905 * so that it can be returned as
4906 * mac_tx_cookie_t to client
4907 */
4908 ringp->s_ring_state |= S_RING_TX_HIWAT;
4909 cookie = (mac_tx_cookie_t)ringp;
4910 ringp->s_ring_hiwat_cnt++;
4911 if (ringp->s_ring_count >
4912 ringp->s_ring_tx_max_q_cnt) {
4913 /* increment freed stats */
4914 ringp->s_ring_drops += cnt;
4915 /*
4916 * b_prev may be set to the fanout hint
4917 * hence can't use freemsg directly
4918 */
4919 mac_drop_chain(mp_chain,
4920 "Tx softring max queue");
4921 DTRACE_PROBE1(tx_queued_hiwat,
4922 mac_soft_ring_t *, ringp);
4923 enqueue = B_FALSE;
4924 }
4925 }
4926 if (enqueue) {
4927 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain,
4928 tail, cnt, sz);
4929 }
4930 }
4931 if (wakeup_worker)
4932 cv_signal(&ringp->s_ring_async);
4933 }
4934 return (cookie);
4935 }
4936
4937
4938 /*
4939 * mac_tx_soft_ring_process
4940 *
4941 * This routine is called when fanning out outgoing traffic among
4942 * multipe Tx rings.
4943 * Note that a soft ring is associated with a h/w Tx ring.
4944 */
4945 mac_tx_cookie_t
mac_tx_soft_ring_process(mac_soft_ring_t * ringp,mblk_t * mp_chain,uint16_t flag,mblk_t ** ret_mp)4946 mac_tx_soft_ring_process(mac_soft_ring_t *ringp, mblk_t *mp_chain,
4947 uint16_t flag, mblk_t **ret_mp)
4948 {
4949 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4950 int cnt;
4951 size_t sz;
4952 mblk_t *tail;
4953 mac_tx_cookie_t cookie = 0;
4954
4955 ASSERT(ringp != NULL);
4956 ASSERT(mp_chain != NULL);
4957 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4958 /*
4959 * The following modes can come here: SRS_TX_BW_FANOUT,
4960 * SRS_TX_FANOUT, SRS_TX_AGGR, SRS_TX_BW_AGGR.
4961 */
4962 ASSERT(MAC_TX_SOFT_RINGS(mac_srs));
4963 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT ||
4964 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT ||
4965 mac_srs->srs_tx.st_mode == SRS_TX_AGGR ||
4966 mac_srs->srs_tx.st_mode == SRS_TX_BW_AGGR);
4967
4968 if (ringp->s_ring_type & ST_RING_WORKER_ONLY) {
4969 /* Serialization mode */
4970
4971 mutex_enter(&ringp->s_ring_lock);
4972 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) {
4973 cookie = mac_tx_sring_enqueue(ringp, mp_chain,
4974 flag, ret_mp);
4975 mutex_exit(&ringp->s_ring_lock);
4976 return (cookie);
4977 }
4978 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
4979 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4980 if (ringp->s_ring_state & (S_RING_BLOCK | S_RING_PROC)) {
4981 /*
4982 * If ring is blocked due to lack of Tx
4983 * descs, just return. Worker thread
4984 * will get scheduled when Tx desc's
4985 * become available.
4986 */
4987 mutex_exit(&ringp->s_ring_lock);
4988 return (cookie);
4989 }
4990 mac_soft_ring_worker_wakeup(ringp);
4991 mutex_exit(&ringp->s_ring_lock);
4992 return (cookie);
4993 } else {
4994 /* Default fanout mode */
4995 /*
4996 * S_RING_BLOCKED is set when underlying NIC runs
4997 * out of Tx descs and messages start getting
4998 * queued. It won't get reset until
4999 * tx_srs_drain() completely drains out the
5000 * messages.
5001 */
5002 mac_tx_stats_t stats;
5003
5004 if (ringp->s_ring_state & S_RING_ENQUEUED) {
5005 /* Tx descs/resources not available */
5006 mutex_enter(&ringp->s_ring_lock);
5007 if (ringp->s_ring_state & S_RING_ENQUEUED) {
5008 cookie = mac_tx_sring_enqueue(ringp, mp_chain,
5009 flag, ret_mp);
5010 mutex_exit(&ringp->s_ring_lock);
5011 return (cookie);
5012 }
5013 /*
5014 * While we were computing mblk count, the
5015 * flow control condition got relieved.
5016 * Continue with the transmission.
5017 */
5018 mutex_exit(&ringp->s_ring_lock);
5019 }
5020
5021 mp_chain = mac_tx_send(ringp->s_ring_tx_arg1,
5022 ringp->s_ring_tx_arg2, mp_chain, &stats);
5023
5024 /*
5025 * Multiple threads could be here sending packets.
5026 * Under such conditions, it is not possible to
5027 * automically set S_RING_BLOCKED bit to indicate
5028 * out of tx desc condition. To atomically set
5029 * this, we queue the returned packet and do
5030 * the setting of S_RING_BLOCKED in
5031 * mac_tx_soft_ring_drain().
5032 */
5033 if (mp_chain != NULL) {
5034 mutex_enter(&ringp->s_ring_lock);
5035 cookie =
5036 mac_tx_sring_enqueue(ringp, mp_chain, flag, ret_mp);
5037 mutex_exit(&ringp->s_ring_lock);
5038 return (cookie);
5039 }
5040 SRS_TX_STATS_UPDATE(mac_srs, &stats);
5041 SOFTRING_TX_STATS_UPDATE(ringp, &stats);
5042
5043 return (0);
5044 }
5045 }
5046