1.\" 2.\" This file and its contents are supplied under the terms of the 3.\" Common Development and Distribution License ("CDDL"), version 1.0. 4.\" You may only use this file in accordance with the terms of version 5.\" 1.0 of the CDDL. 6.\" 7.\" A full copy of the text of the CDDL should have accompanied this 8.\" source. A copy of the CDDL is also available via the Internet at 9.\" http://www.illumos.org/license/CDDL. 10.\" 11.\" 12.\" Copyright 2019 Joyent, Inc. 13.\" Copyright 2020 RackTop Systems, Inc. 14.\" 15.Dd May 11, 2020 16.Dt MAC 9E 17.Os 18.Sh NAME 19.Nm mac , 20.Nm GLDv3 21.Nd MAC networking device driver overview 22.Sh SYNOPSIS 23.In sys/mac_provider.h 24.In sys/mac_ether.h 25.Sh INTERFACE LEVEL 26illumos DDI specific 27.Sh DESCRIPTION 28The 29.Sy MAC 30framework provides a means for implementing high-performance networking 31device drivers. 32It is the successor to the GLD interfaces and is sometimes referred to as the 33GLDv3. 34The remainder of this manual introduces the aspects of writing devices drivers 35that leverage the MAC framework. 36While both the GLDv3 and MAC framework refer to the same thing, in this manual 37page we use the term the 38.Em MAC framework 39to refer to the device driver interface. 40.Pp 41MAC device drivers are character devices. 42They define the standard 43.Xr _init 9E , 44.Xr _fini 9E , 45and 46.Xr _info 9E 47entry points to initialize the module, as well as 48.Xr dev_ops 9S 49and 50.Xr cb_ops 9S 51structures. 52.Pp 53The main interface with MAC is through a series of callbacks defined in 54a 55.Xr mac_callbacks 9S 56structure. 57These callbacks control all the aspects of the device. 58They range from sending data, getting and setting of properties, controlling mac 59address filters, and also managing promiscuous mode. 60.Pp 61The MAC framework takes care of many aspects of the device driver's 62management. 63A device that uses the MAC framework does not have to worry about creating 64device nodes or implementing 65.Xr open 9E 66or 67.Xr close 9E 68routines. 69In addition, all of the work to interact with 70.Xr dlpi 7P 71is taken care of automatically and transparently. 72.Ss Initializing MAC Support 73For a device to be used in the framework, it must register with the 74framework and take specific actions during 75.Xr _init 9E , 76.Xr attach 9E , 77.Xr detach 9E , 78and 79.Xr _fini 9E . 80.Pp 81All device drivers have to define a 82.Xr dev_ops 9S 83structure which is pointed to by a 84.Xr modldrv 9S 85structure and the corresponding NULL-terminated 86.Xr modlinkage 9S 87structure. 88The 89.Xr dev_ops 9S 90structure should have a 91.Xr cb_ops 9S 92structure defined for it; however, it does not need to implement any of 93the standard 94.Xr cb_ops 9S 95entry points. 96.Pp 97Normally, in a driver's 98.Xr _init 9E 99entry point, it passes its 100.Sy modlinkage 101structure directly to 102.Xr mod_install 9F . 103To properly register with MAC, the driver must call 104.Xr mac_init_ops 9F 105before it calls 106.Xr mod_install 9F . 107If for some reason the 108.Xr mod_install 9F 109function fails, then the driver must be removed by a call to 110.Xr mac_fini_ops 9F . 111.Pp 112Conversely, in the driver's 113.Xr _fini 9E 114routine, it should call 115.Xr mac_fini_ops 9F 116after it successfully calls 117.Xr mod_remove 9F . 118For an example of how to use the 119.Xr mac_init_ops 9F 120and 121.Xr mac_fini_ops 9F 122functions, see the examples section in 123.Xr mac_init_ops 9F . 124.Ss Registering with MAC 125Every instance of a device should register separately with MAC. 126To register with MAC, a driver must allocate a 127.Xr mac_register 9S 128structure, fill it in, and then call 129.Xr mac_register 9F . 130The 131.Sy mac_register_t 132structure contains information about the device and all of the required 133function pointers that will be used as callbacks by the framework. 134.Pp 135These steps should all be taken during a device's 136.Xr attach 9E 137entry point. 138It is recommended that the driver perform this sequence of steps after the 139device has finished its initialization of the chipset and interrupts, though 140interrupts should not be enabled at that point. 141After it calls 142.Xr mac_register 9F 143it will start receiving callbacks from the MAC framework. 144.Pp 145To allocate the registration structure, the driver should call 146.Xr mac_alloc 9F . 147Device drivers should generally always pass the symbol 148.Sy MAC_VERSION 149as the argument to 150.Xr mac_alloc 9F . 151Upon successful completion, the driver will receive a 152.Sy mac_register_t 153structure which it should fill in. 154The structure and its members are documented in 155.Xr mac_register 9S . 156.Pp 157The 158.Xr mac_callbacks 9S 159structure is not allocated as a part of the 160.Xr mac_register 9S 161structure. 162In general, device drivers declare this statically. 163See the 164.Sx MAC Callbacks 165section for more information on how to fill it out. 166.Pp 167Once the structure has been filled in, the driver should call 168.Xr mac_register 9F 169to register itself with MAC. 170The handle that it uses to register with should be part of the driver's soft 171state. 172It will be used in various other support functions and callbacks. 173.Pp 174If the call is successful, then the device driver 175should enable interrupts and finish any other initialization required. 176If the call to 177.Xr mac_register 9F 178failed, then it should unwind its initialization and should return 179.Sy DDI_FAILURE 180from its 181.Xr attach 9E 182routine. 183.Pp 184The driver does not need to hold onto an allocated 185.Xr mac_register 9S 186structure after it has called the 187.Xr mac_register 9F 188function. 189Whether the 190.Xr mac_register 9F 191function returns successfully or not, the driver may free its 192.Xr mac_register 9S 193structure by calling the 194.Xr mac_free 9F 195function. 196.Ss MAC Callbacks 197The MAC framework interacts with a device driver through a series of 198callbacks. 199These callbacks are described in their individual manual pages and the 200collection of callbacks is indicated in the 201.Xr mac_callbacks 9S 202manual page. 203This section does not focus on the specific functions, but rather on 204interactions between them and the rest of the device driver framework. 205.Pp 206A device driver should make no assumptions about when the various 207callbacks will be called and whether or not they will be called 208simultaneously. 209For example, a device driver may be asked to transmit data through a call to its 210.Xr mc_tx 9E 211entry point while it is being asked to get a device property through a 212call to its 213.Xr mc_getprop 9E 214entry point. 215As such, while some calls may be serialized to the device, such as setting 216properties, the device driver should always presume that all of its data needs 217to be protected with locks. 218While the device is holding locks, it is safe for it call the following MAC 219routines: 220.Bl -bullet -offset indent -compact 221.It 222.Xr mac_hcksum_get 9F 223.It 224.Xr mac_hcksum_set 9F 225.It 226.Xr mac_lso_get 9F 227.It 228.Xr mac_maxsdu_update 9F 229.It 230.Xr mac_prop_info_set_default_link_flowctrl 9F 231.It 232.Xr mac_prop_info_set_default_str 9F 233.It 234.Xr mac_prop_info_set_default_uint8 9F 235.It 236.Xr mac_prop_info_set_default_uint32 9F 237.It 238.Xr mac_prop_info_set_default_uint64 9F 239.It 240.Xr mac_prop_info_set_perm 9F 241.It 242.Xr mac_prop_info_set_range_uint32 9F 243.El 244.Pp 245Any other MAC related routines should not be called with locks held, 246such as 247.Xr mac_link_update 9F 248or 249.Xr mac_rx 9F . 250Other routines in the DDI may be called while locks are held; however, 251device driver writers should be careful about calling blocking routines 252while locks are held or in interrupt context, though it is generally 253legal to do so. 254.Ss Receiving Data 255A device driver will often receive data through the means of an 256interrupt. 257When that interrupt occurs, the device driver will receive one or more frames 258with optional metadata. 259Often each frame has a corresponding descriptor which has information about 260whether or not there were errors or whether or not the device successfully 261checksummed the packet. 262In addition to the per-packet flow described below, there are certain 263requirements that drivers must adhere to when programming the hardware 264to receive data. 265See the section 266.Sx RECEIVE DESCRIPTOR LAYOUT 267for more information. 268.Pp 269During a single interrupt, a device driver should process a fixed number 270of frames. 271For each frame the device driver should: 272.Bl -enum -offset indent 273.It 274First check whether or not the frame has errors. 275If errors were detected, then the frame should not be sent to the operating 276system. 277It is recommended that devices keep kstats (see 278.Xr kstat_create 9F 279for more information) and bump the counter whenever such an error is 280detected. 281If the device distinguishes between the types of errors, then separate kstats 282for each class of error are recommended. 283See the 284.Sx STATISTICS 285section for more information on the various error cases that should be 286considered. 287.It 288Once the frame has been determined to be valid, the device driver should 289transform the frame into a 290.Xr mblk 9S . 291See the section 292.Sx MBLKS AND DMA 293for more information on how to transform and prepare a message block. 294.It 295If the device supports hardware checksumming (see the 296.Sx CAPABILITIES 297section for more information on checksumming), then the device driver 298should set the corresponding checksumming information with a call to 299.Xr mac_hcksum_set 9F . 300.It 301It should then append this new message block to the 302.Em end 303of the message block chain, linking it to the 304.Sy b_next 305pointer. 306It is vitally important that all the frames be chained in the order that they 307were received. 308If the device driver mistakenly reorders frames, then it may cause performance 309impacts in the TCP stack and potentially impact application correctness. 310.El 311.Pp 312Once all the frames have been processed and assembled, the device driver 313should deliver them to the rest of the operating system by calling 314.Xr mac_rx 9F . 315The device driver should try to give as many mblk_t structures to the 316system at once. 317It 318.Em should not 319call 320.Xr mac_rx 9F 321once for every assembled mblk_t. 322.Pp 323The device driver must not hold any locks across the call to 324.Xr mac_rx 9F . 325When this function is called, received data will be pushed through the 326networking stack and some replies may be generated and given to the 327driver to send out. 328.Pp 329It is not the device driver's responsibility to determine whether or not 330the system can keep up with a driver's delivery rate of frames. 331The rest of the networking stack will handle issues related to keeping up 332appropriately and ensure that kernel memory is not exhausted by packets 333that are not being processed. 334.Pp 335Finally, the device driver should make sure that any other housekeeping 336activities required for the ring are taken care of such that more data 337can be received. 338.Ss Transmitting Data and Back Pressure 339A device driver will be asked to transmit a message block chain by 340having it's 341.Xr mc_tx 9E 342entry point called. 343While the driver is processing the message blocks, it may run out of resources. 344For example, a transmit descriptor ring may become full. 345At that point, the device driver should return the remaining unprocessed frames. 346The act of returning frames indicates that the device has asserted flow control. 347Once this has been done, no additional calls will be made to the 348driver's transmit entry point and the back pressure will be propagated 349throughout the rest of the networking stack. 350.Pp 351At some point in the future when resources have become available again, 352for example after an interrupt indicating that some portion of the 353transmit ring has been sent, then the device driver must notify the 354system that it can continue transmission. 355To do this, the driver should call 356.Xr mac_tx_update 9F . 357After that point, the driver will receive calls to its 358.Xr mc_tx 9E 359entry point again. 360As mentioned in the section on callbacks, the device driver should avoid holding 361any particular locks across the call to 362.Xr mac_tx_update 9F . 363.Ss Interrupt Coalescing 364For devices operating at higher data rates, interrupt coalescing is an 365important part of a well functioning device and may impact the 366performance of the device. 367Not all devices support interrupt coalescing. 368If interrupt coalescing is supported on the device, it is recommended that 369device driver writers provide private properties for their device to control the 370interrupt coalescing rate. 371This will make it much easier to perform experiments and observe the impact of 372different interrupt rates on the rest of the system. 373.Ss MAC Address Filter Management 374The MAC framework will attempt to use as many MAC address filters as a 375device has. 376To program a multicast address filter, the driver's 377.Xr mc_multicst 9E 378entry point will be called. 379If the device driver runs out of filters, it should not take any special action 380and just return the appropriate error as documented in the corresponding manual 381pages for the entry points. 382The framework will ensure that the device is placed in promiscuous mode 383if it needs to. 384.Ss Link Updates 385It is the responsibility of the device driver to keep track of the 386data link's state. 387Many devices provide a means of receiving an interrupt when the state of the 388link changes. 389When such a change happens, the driver should update its internal data 390structures and then call 391.Xr mac_link_update 9F 392to inform the MAC layer that this has occurred. 393If the device driver does not properly inform the system about link changes, 394then various features like link aggregations and other mechanisms that leverage 395the link state will not work correctly. 396.Ss Link Speed and Auto-negotiation 397Many networking devices support more than one possible speed that they 398can operate at. 399The selection of a speed is often performed through 400.Em auto-negotiation , 401though some devices allow the user to control what speeds are advertised 402and used. 403.Pp 404Logically, there are two different sets of things that the device driver 405needs to keep track of while it's operating: 406.Bl -enum 407.It 408The supported speeds in hardware. 409.It 410The enabled speeds from the user. 411.El 412.Pp 413By default, when a link first comes up, the device driver should 414generally configure the link to support the common set of speeds and 415perform auto-negotiation. 416.Pp 417A user can control what speeds a device advertises via auto-negotiation 418and whether or not it performs auto-negotiation at all by using a series 419of properties that have 420.Sy _EN_ 421in the name. 422These are read/write properties and there is one for each speed supported in the 423operating system. 424For a full list of them, see the 425.Sx PROPERTIES 426section. 427.Pp 428In addition to these properties, there is a corresponding set of 429properties with 430.Sy _ADV_ 431in the name. 432These are similar to the 433.Sy _EN_ 434family of properties, but they are read-only and indicate what the 435device has actually negotiated. 436While they are generally similar to the 437.Sy _EN_ 438family of properties, they may change depending on power settings. 439See the 440.Sy Ethernet Link Properties 441section in 442.Xr dladm 1M 443for more information. 444.Pp 445It's worth discussing how these different values get used throughout the 446different entry points. 447The first entry point to consider is the 448.Xr mc_propinfo 9E 449entry point. 450For a given speed, the driver should consult whether or not the hardware 451supports this speed. 452If it does, it should fill in the default value that the hardware takes and 453whether or not the property is writable. 454The properties should also be updated to indicate whether or not it is writable. 455This holds for both the 456.Sy _EN_ 457and 458.Sy _ADV_ 459family of properties. 460.Pp 461The next entry point is 462.Xr mc_getprop 9E . 463Here, the device should first consult whether the given speed is 464supported. 465If it is not, then the driver should return 466.Er ENOTSUP . 467If it does, then it should return the current value of the property. 468.Pp 469The last property endpoint is the 470.Xr mc_setprop 9E 471entry point. 472Here, the same logic applies. 473Before the driver considers whether or not the property is writable, it should 474first check whether or not it's a supported property. 475If it's not, then it should return 476.Er ENOTSUP . 477Otherwise, it should proceed to check whether the property is writable, 478and if it is and a valid value, then it should update the property and 479restart the link's negotiation. 480.Pp 481Finally, there is the 482.Xr mc_getstat 9E 483entry point. 484Several of the statistics that are queried relate to auto-negotiation and 485hardware capabilities. 486When a statistic relates to the hardware supporting a given speed, the 487.Sy _EN_ 488properties should be ignored. 489The only thing that should be consulted is what the hardware itself supports. 490Otherwise, the statistics should look at what is currently being advertised by 491the device. 492.Ss Unregistering from MAC 493During a driver's 494.Xr detach 9E 495routine, it should unregister the device instance from MAC by calling 496.Xr mac_unregister 9F 497on the handle that it originally called it on. 498If the call to 499.Xr mac_unregister 9F 500failed, then the device is likely still in use and the driver should 501fail the call to 502.Xr detach 9E . 503.Ss Interacting with Devices 504Administrators always interact with devices through the 505.Xr dladm 1M 506command line interface. 507The state of devices such as whether the link is considered 508.Sy up 509or 510.Sy down , 511various link properties such as the 512.Sy MTU , 513.Sy auto-negotiation 514state, 515and 516.Sy flow control 517state, 518are all exposed. 519It is also the preferred way that these properties are set and configured. 520.Pp 521While device tunables may be presented in a 522.Xr driver.conf 4 523file, it is recommended instead to expose such things through 524.Xr dladm 1M 525private properties, whether explicitly documented or not. 526.Sh CAPABILITIES 527Capabilities in the MAC Framework are optional features that a device 528supports which indicate various hardware features that the device 529supports. 530The two current capabilities that the system supports are related to being able 531to hardware perform large send offloads (LSO), often also known as TCP 532segmentation and the ability for hardware to calculate and verify the checksums 533present in IPv4, IPV6, and protocol headers such as TCP and UDP. 534.Pp 535The MAC framework will query a device for support of a capability 536through the 537.Xr mc_getcapab 9E 538function. 539Each capability has its own constant and may have corresponding data that goes 540along with it and a specific structure that the device is required to fill in. 541Note, the set of capabilities changes over time and there are also private 542capabilities in the system. 543Several of the capabilities are used in the implementation of the MAC framework. 544Others, like 545.Sy MAC_CAPAB_RINGS , 546represent feature that have not been stabilized and thus both API and binary 547compatibility for them is not guaranteed. 548It is important that the device driver handles unknown capabilities correctly. 549For more information, see 550.Xr mc_getcapab 9E . 551.Pp 552The following capabilities are 553stable and defined in the system: 554.Ss MAC_CAPAB_HCKSUM 555The 556.Sy MAC_CAPAB_HCKSUM 557capability indicates to the system that the device driver supports some 558amount of checksumming. 559The specific data for this capability is a pointer to a 560.Sy uint32_t . 561To indicate no support for any kind of checksumming, the driver should 562either set this value to zero or simply return that it doesn't support 563the capability. 564.Pp 565Note, the values that the driver declares in this capability indicate 566what it can do when it transmits data. 567If the driver can only verify checksums when receiving data, then it should not 568indicate that it supports this capability. 569The following set of flags may be combined through a bitwise inclusive OR: 570.Bl -tag -width Ds 571.It Sy HCKSUM_INET_PARTIAL 572This indicates that the hardware can calculate a partial checksum for 573both IPv4 and IPv6; however, it requires the pseudo-header checksum be 574calculated for it. 575The pseudo-header checksum will be available for the mblk_t when calling 576.Xr mac_hcksum_get 9F . 577Note this does not imply that the hardware is capable of calculating the 578IPv4 header checksum. 579That should be indicated with the 580.Sy HCKSUM_IPHDRCKSUM flag. 581.It Sy HCKSUM_INET_FULL_V4 582This indicates that the hardware will fully calculate the L4 checksum 583for outgoing IPv4 packets and does not require a pseudo-header checksum. 584Note this does not imply that the hardware is capable of calculating the 585IPv4 header checksum. 586That should be indicated with the 587.Sy HCKSUM_IPHDRCKSUM . 588.It Sy HCKSUM_INET_FULL_V6 589This indicates that the hardware will fully calculate the L4 checksum 590for outgoing IPv6 packets and does not require a pseudo-header checksum. 591.It Sy HCKSUM_IPHDRCKSUM 592This indicates that the hardware supports calculating the checksum for 593the IPv4 header itself. 594.El 595.Pp 596When in a driver's transmit function, the driver will be processing a 597single frame. 598It should call 599.Xr mac_hcksum_get 9F 600to see what checksum flags are set on it. 601Note that the flags that are set on it are different from the ones described 602above and are documented in its manual page. 603These flags indicate how the driver is expected to program the hardware and what 604checksumming is required. 605Not all frames will require hardware checksumming or will ask the hardware to 606checksum it. 607.Pp 608If a driver supports offloading the receive checksum and verification, 609it should check to see what the hardware indicated was verified. 610The driver should then call 611.Xr mac_hcksum_set 9F . 612The flags used are different from the ones above and are discussed in 613detail in the 614.Xr mac_hcksum_set 9F 615manual page. 616If there is no checksum information available or the driver does not support 617checksumming, then it should simply not call 618.Xr mac_hcksum_set 9F . 619.Pp 620Note that the checksum flags should be set on the first 621mblk_t that makes up a given message. 622In other words, if multiple mblk_t structures are linked together by the 623.Sy b_cont 624member to describe a single frame, then it should only be called on the 625first mblk_t of that set. 626However, each distinct message should have the checksum bits set on it, if 627applicable. 628In other words, each mblk_t that is linked together by the 629.Sy b_next 630pointer may have checksum flags set. 631.Pp 632It is recommended that device drivers provide a private property or 633.Xr driver.conf 4 634property to control whether or not checksumming is enabled for both rx 635and tx; however, the default disposition is recommended to be enabled 636for both. 637This way if hardware bugs are found in the checksumming implementation, they can 638be disabled without requiring software updates. 639The transmit property should be checked when determining how to reply to 640.Xr mc_getcapab 9E 641and the receive property should be checked in the context of the receive 642function. 643.Ss MAC_CAPAB_LSO 644The 645.Sy MAC_CAPAB_LSO 646capability indicates that the driver supports various forms of large 647send offload (LSO). 648The private data is a pointer to a 649.Sy mac_capab_lso_t 650structure. 651At the moment, LSO support is limited to TCP inside of IPv4. 652This structure has the following members which are used to indicate 653various types of LSO support. 654.Bd -literal -offset indent 655t_uscalar_t lso_flags; 656lso_basic_tcp_ivr4_t lso_basic_tcp_ipv4; 657.Ed 658.Pp 659The 660.Sy lso_flags 661member is used to indicate which members are valid and should be 662considered. 663Each flag represents a different form of LSO. 664The member should be set to the bitwise inclusive OR of the following values: 665.Bl -tag -width Dv -offset indent 666.It Sy LSO_TX_BASIC_TCP_IPV4 667This indicates hardware support for performing TCP segmentation 668offloading over IPv4. 669When this flag is set, the 670.Sy lso_basic_tcp_ipv4 671member must be filled in. 672.El 673.Pp 674The 675.Sy lso_basic_tcp_ipv4 676member is a structure with the following members: 677.Bd -literal -offset indent 678t_uscalar_t lso_max 679.Ed 680.Bd -filled -offset indent 681The 682.Sy lso_max 683member should be set to the maximum size of the TCP data 684payload that can be offloaded to the hardware. 685.Ed 686.Pp 687Like with checksumming, it is recommended that driver writers provide a 688means for disabling the support of LSO even if it is enabled by default. 689This deals with the case where issues that pop up for LSO may be worked 690around without requiring additional driver work. 691.Sh PROPERTIES 692Properties in the MAC framework represent aspects of a link. 693These include things like the link's current state and MTU. 694Many of the properties in the system are focused around auto-negotiation and 695controlling what link speeds are advertised. 696Information about properties is covered by three different device entry points. 697The 698.Xr mc_propinfo 9E 699entry point obtains metadata about the property. 700The 701.Xr mc_getprop 9E 702entry point obtains the property. 703The 704.Xr mc_setprop 9E 705entry point updates the property to a new value. 706.Pp 707Many of the properties listed below are read-only. 708Each property indicates whether it's read-only or it's read/write. 709However, driver writers may not implement the ability to set all writable 710properties. 711Many of these depend on the card itself. 712In particular, all properties that relate to auto-negotiation and are read/write 713may not be updated if the hardware in question does not support toggling what 714link speeds are auto-negotiated. 715While copper Ethernet often does not have this restriction, it often exists with 716various fiber standards and phys. 717.Pp 718The following properties are the subset of MAC framework properties that 719driver writers should be aware of and handle. 720While other properties exist in the system, driver writers should always return 721an error when a property not listed below is encountered. 722See 723.Xr mc_getprop 9E 724and 725.Xr mc_setprop 9E 726for more information on how to handle them. 727.Bl -hang -width Ds 728.It Sy MAC_PROP_DUPLEX 729.Bd -filled -compact 730Type: 731.Sy link_duplex_t | 732Permissions: 733.Sy Read-Only 734.Ed 735.Pp 736The 737.Sy MAC_PROP_DUPLEX 738property is used to indicate whether or not the link is duplex. 739A duplex link may have traffic flowing in both directions at the same time. 740The 741.Sy link_duplex_t 742is an enumeration which may be set to any of the following values: 743.Bl -tag -width Ds 744.It Sy LINK_DUPLEX_UNKNOWN 745The current state of the link is unknown. 746This may be because the link has not negotiated to a specific speed or it is 747down. 748.It Sy LINK_DUPLEX_HALF 749The link is running at half duplex. 750Communication may travel in only one direction on the link at a given time. 751.It Sy LINK_DUPLEX_FULL 752The link is running at full duplex. 753Communication may travel in both directions on the link simultaneously. 754.El 755.It Sy MAC_PROP_SPEED 756.Bd -filled -compact 757Type: 758.Sy uint64_t | 759Permissions: 760.Sy Read-Only 761.Ed 762.Pp 763The 764.Sy MAC_PROP_SPEED 765property stores the current link speed in bits per second. 766A link that is running at 100 MBit/s would store the value 100000000ULL. 767A link that is running at 40 Gbit/s would store the value 40000000000ULL. 768.It Sy MAC_PROP_STATUS 769.Bd -filled -compact 770Type: 771.Sy link_state_t | 772Permissions: 773.Sy Read-Only 774.Ed 775.Pp 776The 777.Sy MAC_PROP_STATUS 778property is used to indicate the current state of the link. 779It indicates whether the link is up or down. 780The 781.Sy link_state_t 782is an enumeration which may be set to any of the following values: 783.Bl -tag -width Ds 784.It Sy LINK_STATE_UNKNOWN 785The current state of the link is unknown. 786This may be because the driver's 787.Xr mc_start 9E 788endpoint has not been called so it has not attempted to start the link. 789.It Sy LINK_STATE_DOWN 790The link is down. 791This may be because of a negotiation problem, a cable problem, or some other 792device specific issue. 793.It Sy LINK_STATE_UP 794The link is up. 795If auto-negotiation is in use, it should have completed. 796Traffic should be able to flow over the link, barring other issues. 797.El 798.It Sy MAC_PROP_AUTONEG 799.Bd -filled -compact 800Type: 801.Sy uint8_t | 802Permissions: 803.Sy Read/Write 804.Ed 805.Pp 806The 807.Sy MAC_PROP_AUTONEG 808property indicates whether or not the device is currently configured to 809perform auto-negotiation. 810A value of 811.Sy 0 812indicates that auto-negotiation is disabled. 813A 814.Sy non-zero 815value indicates that auto-negotiation is enabled. 816Devices should generally default to enabling auto-negotiation. 817.Pp 818When getting this property, the device driver should return the current 819state. 820When setting this property, if the device supports operating in the requested 821mode, then the device driver should reset the link to negotiate to the new speed 822after updating any internal registers. 823.It Sy MAC_PROP_MTU 824.Bd -filled -compact 825Type: 826.Sy uint32_t | 827Permissions: 828.Sy Read/Write 829.Ed 830.Pp 831The 832.Sy MAC_PROP_MTU 833property determines the maximum transmission unit (MTU). 834This indicates the maximum size packet that the device can transmit, ignoring 835its own headers. 836For an Ethernet device, this would exclude the size of the Ethernet header and 837any VLAN headers that would be placed. 838It is up to the driver to ensure that any MTU values that it accepts when adding 839in its margin and header sizes does not exceed its maximum frame size. 840.Pp 841By default, drivers for Ethernet should initialize this value and the 842MTU to 843.Sy 1500 . 844When getting this property, the driver should return its current 845recorded MTU. 846When setting this property, the driver should first validate that it is within 847the device's valid range and then it must call 848.Xr mac_maxsdu_update 9F . 849Note that the call may fail. 850If the call completes successfully, the driver should update the hardware with 851the new value of the MTU and perform any other work needed to handle it. 852.Pp 853If the device does not support changing the MTU after the device's 854.Xr mc_start 9E 855entry point has been called, then driver writers should return 856.Er EBUSY . 857.It Sy MAC_PROP_FLOWCTRL 858.Bd -filled -compact 859Type: 860.Sy link_flowctrl_t | 861Permissions: 862.Sy Read/Write 863.Ed 864.Pp 865The 866.Sy MAC_PROP_FLOWCTRL 867property manages the configuration of pause frames as part of Ethernet 868flow control. 869Note, this only describes what this device will advertise. 870What is actually enabled may be different and is subject to the rules of 871auto-negotiation. 872The 873.Sy link_flowctrl_t 874is an enumeration that may be set to one of the following values: 875.Bl -tag -width Ds 876.It Sy LINK_FLOWCTRL_NONE 877Flow control is disabled. 878No pause frames should be generated or honored. 879.It Sy LINK_FLOWCTRL_RX 880The device can receive pause frames; however, it should not generate 881them. 882.It Sy LINK_FLOWCTRL_TX 883The device can generate pause frames; however, it does not support 884receiving them. 885.It Sy LINK_FLOWCTRL_BI 886The device supports both sending and receiving pause frames. 887.El 888.Pp 889When getting this property, the device driver should return the way that 890it has configured the device, not what the device has actually 891negotiated. 892When setting the property, it should update the hardware and allow the link to 893potentially perform auto-negotiation again. 894.It Sy MAC_PROP_EN_FEC_CAP 895.Bd -filled -compact 896Type: 897.Sy link_fec_t | 898Permissions: 899.Sy Read/Write 900.Ed 901.Pp 902The 903.Sy MAC_PROP_EN_FEC_CAP 904property indicates which Forward Error Correction (FEC) code is advertised 905by the device. 906.Pp 907The 908.Sy link_fec_t 909is an enumeration that may be a combination of the following bit values: 910.Bl -tag -width Ds 911.It Sy LINK_FEC_NONE 912No FEC over the link. 913.It Sy LINK_FEC_AUTO 914The FEC coding to use is auto-negotiated, 915.Sy LINK_FEC_AUTO 916cannot be set along with any of the other values. 917This is the default setting the device driver should use. 918.It Sy LINK_FEC_RS 919The link may use Reed-Solomon FEC coding. 920.It Sy LINK_FEC_BASE_R 921The link may use Base-R coding, also common referred to as FireCode. 922.El 923.Pp 924When setting the property, it should update the hardware with the requested, or 925combination of requested codings. 926If a particular combination of codings is not supported by the hardware, 927the device driver should return 928.Er EINVAL . 929When retrieving this property, the device driver should return the current 930value of the property. 931.It Sy MAC_PROP_ADV_FEC_CAP 932.Bd -filled -compact 933Type: 934.Sy link_fec_t | 935Permissions: 936.Sy Read-Only 937.Ed 938.Pp 939The 940.Sy MAC_PROP_ADV_FEC_CAP 941has the same values as 942.Sy MAC_PROP_EN_FEC_CAP . 943The property indicates which Forward Error Correction (FEC) code has been 944negotiated over the link. 945.El 946.Pp 947The remaining properties are all about various auto-negotiation link 948speeds. 949They fall into two different buckets: properties with 950.Sy _ADV_ 951in the name and properties with 952.Sy _EN_ 953in the name. 954For any given supported speed, there is one of each. 955The 956.Sy _EN_ 957set of properties are read/write properties that control what should be 958advertised by the device. 959When these are retrieved, they should return the current value of the property. 960When they are set, they should change how the hardware advertises the specific 961speed and trigger any kind of link reset and auto-negotiation, if enabled, to 962occur. 963.Pp 964The 965.Sy _ADV_ 966set of properties are read-only properties. 967They are meant to reflect what has actually been negotiated. 968These may be different from the 969.Sy _EN_ 970family of properties, especially when different power management 971settings are at play. 972.Pp 973See the 974.Sx Link Speed and Auto-negotiation 975section for more information. 976.Pp 977The properties are ordered in increasing link speed: 978.Bl -hang -width Ds 979.It Sy MAC_PROP_ADV_10HDX_CAP 980.Bd -filled -compact 981Type: 982.Sy uint8_t | 983Permissions: 984.Sy Read-Only 985.Ed 986.Pp 987The 988.Sy MAC_PROP_ADV_10HDX_CAP 989property describes whether or not 10 Mbit/s half-duplex support is 990advertised. 991.It Sy MAC_PROP_EN_10HDX_CAP 992.Bd -filled -compact 993Type: 994.Sy uint8_t | 995Permissions: 996.Sy Read/Write 997.Ed 998.Pp 999The 1000.Sy MAC_PROP_EN_10HDX_CAP 1001property describes whether or not 10 Mbit/s half-duplex support is 1002enabled. 1003.It Sy MAC_PROP_ADV_10FDX_CAP 1004.Bd -filled -compact 1005Type: 1006.Sy uint8_t | 1007Permissions: 1008.Sy Read-Only 1009.Ed 1010.Pp 1011The 1012.Sy MAC_PROP_ADV_10FDX_CAP 1013property describes whether or not 10 Mbit/s full-duplex support is 1014advertised. 1015.It Sy MAC_PROP_EN_10FDX_CAP 1016.Bd -filled -compact 1017Type: 1018.Sy uint8_t | 1019Permissions: 1020.Sy Read/Write 1021.Ed 1022.Pp 1023The 1024.Sy MAC_PROP_EN_10FDX_CAP 1025property describes whether or not 10 Mbit/s full-duplex support is 1026enabled. 1027.It Sy MAC_PROP_ADV_100HDX_CAP 1028.Bd -filled -compact 1029Type: 1030.Sy uint8_t | 1031Permissions: 1032.Sy Read-Only 1033.Ed 1034.Pp 1035The 1036.Sy MAC_PROP_ADV_100HDX_CAP 1037property describes whether or not 100 Mbit/s half-duplex support is 1038advertised. 1039.It Sy MAC_PROP_EN_100HDX_CAP 1040.Bd -filled -compact 1041Type: 1042.Sy uint8_t | 1043Permissions: 1044.Sy Read/Write 1045.Ed 1046.Pp 1047The 1048.Sy MAC_PROP_EN_100HDX_CAP 1049property describes whether or not 100 Mbit/s half-duplex support is 1050enabled. 1051.It Sy MAC_PROP_ADV_100FDX_CAP 1052.Bd -filled -compact 1053Type: 1054.Sy uint8_t | 1055Permissions: 1056.Sy Read-Only 1057.Ed 1058.Pp 1059The 1060.Sy MAC_PROP_ADV_100FDX_CAP 1061property describes whether or not 100 Mbit/s full-duplex support is 1062advertised. 1063.It Sy MAC_PROP_EN_100FDX_CAP 1064.Bd -filled -compact 1065Type: 1066.Sy uint8_t | 1067Permissions: 1068.Sy Read/Write 1069.Ed 1070.Pp 1071The 1072.Sy MAC_PROP_EN_100FDX_CAP 1073property describes whether or not 100 Mbit/s full-duplex support is 1074enabled. 1075.It Sy MAC_PROP_ADV_100T4_CAP 1076.Bd -filled -compact 1077Type: 1078.Sy uint8_t | 1079Permissions: 1080.Sy Read-Only 1081.Ed 1082.Pp 1083The 1084.Sy MAC_PROP_ADV_100T4_CAP 1085property describes whether or not 100 Mbit/s Ethernet using the 1086100BASE-T4 standard is 1087advertised. 1088.It Sy MAC_PROP_EN_100T4_CAP 1089.Bd -filled -compact 1090Type: 1091.Sy uint8_t | 1092Permissions: 1093.Sy Read/Write 1094.Ed 1095.Pp 1096The 1097.Sy MAC_PROP_ADV_100T4_CAP 1098property describes whether or not 100 Mbit/s Ethernet using the 1099100BASE-T4 standard is 1100enabled. 1101.It Sy MAC_PROP_ADV_1000HDX_CAP 1102.Bd -filled -compact 1103Type: 1104.Sy uint8_t | 1105Permissions: 1106.Sy Read-Only 1107.Ed 1108.Pp 1109The 1110.Sy MAC_PROP_ADV_1000HDX_CAP 1111property describes whether or not 1 Gbit/s half-duplex support is 1112advertised. 1113.It Sy MAC_PROP_EN_1000HDX_CAP 1114.Bd -filled -compact 1115Type: 1116.Sy uint8_t | 1117Permissions: 1118.Sy Read/Write 1119.Ed 1120.Pp 1121The 1122.Sy MAC_PROP_EN_1000HDX_CAP 1123property describes whether or not 1 Gbit/s half-duplex support is 1124enabled. 1125.It Sy MAC_PROP_ADV_1000FDX_CAP 1126.Bd -filled -compact 1127Type: 1128.Sy uint8_t | 1129Permissions: 1130.Sy Read-Only 1131.Ed 1132.Pp 1133The 1134.Sy MAC_PROP_ADV_1000FDX_CAP 1135property describes whether or not 1 Gbit/s full-duplex support is 1136advertised. 1137.It Sy MAC_PROP_EN_1000FDX_CAP 1138.Bd -filled -compact 1139Type: 1140.Sy uint8_t | 1141Permissions: 1142.Sy Read/Write 1143.Ed 1144.Pp 1145The 1146.Sy MAC_PROP_EN_1000FDX_CAP 1147property describes whether or not 1 Gbit/s full-duplex support is 1148enabled. 1149.It Sy MAC_PROP_ADV_2500FDX_CAP 1150.Bd -filled -compact 1151Type: 1152.Sy uint8_t | 1153Permissions: 1154.Sy Read-Only 1155.Ed 1156.Pp 1157The 1158.Sy MAC_PROP_ADV_2500FDX_CAP 1159property describes whether or not 2.5 Gbit/s full-duplex support is 1160advertised. 1161.It Sy MAC_PROP_EN_2500FDX_CAP 1162.Bd -filled -compact 1163Type: 1164.Sy uint8_t | 1165Permissions: 1166.Sy Read/Write 1167.Ed 1168.Pp 1169The 1170.Sy MAC_PROP_EN_2500FDX_CAP 1171property describes whether or not 2.5 Gbit/s full-duplex support is 1172enabled. 1173.It Sy MAC_PROP_ADV_5000FDX_CAP 1174.Bd -filled -compact 1175Type: 1176.Sy uint8_t | 1177Permissions: 1178.Sy Read-Only 1179.Ed 1180.Pp 1181The 1182.Sy MAC_PROP_ADV_5000FDX_CAP 1183property describes whether or not 5.0 Gbit/s full-duplex support is 1184advertised. 1185.It Sy MAC_PROP_EN_5000FDX_CAP 1186.Bd -filled -compact 1187Type: 1188.Sy uint8_t | 1189Permissions: 1190.Sy Read/Write 1191.Ed 1192.Pp 1193The 1194.Sy MAC_PROP_EN_5000FDX_CAP 1195property describes whether or not 5.0 Gbit/s full-duplex support is 1196enabled. 1197.It Sy MAC_PROP_ADV_10GFDX_CAP 1198.Bd -filled -compact 1199Type: 1200.Sy uint8_t | 1201Permissions: 1202.Sy Read-Only 1203.Ed 1204.Pp 1205The 1206.Sy MAC_PROP_ADV_10GFDX_CAP 1207property describes whether or not 10 Gbit/s full-duplex support is 1208advertised. 1209.It Sy MAC_PROP_EN_10GFDX_CAP 1210.Bd -filled -compact 1211Type: 1212.Sy uint8_t | 1213Permissions: 1214.Sy Read/Write 1215.Ed 1216.Pp 1217The 1218.Sy MAC_PROP_EN_10GFDX_CAP 1219property describes whether or not 10 Gbit/s full-duplex support is 1220enabled. 1221.It Sy MAC_PROP_ADV_40GFDX_CAP 1222.Bd -filled -compact 1223Type: 1224.Sy uint8_t | 1225Permissions: 1226.Sy Read-Only 1227.Ed 1228.Pp 1229The 1230.Sy MAC_PROP_ADV_40GFDX_CAP 1231property describes whether or not 40 Gbit/s full-duplex support is 1232advertised. 1233.It Sy MAC_PROP_EN_40GFDX_CAP 1234.Bd -filled -compact 1235Type: 1236.Sy uint8_t | 1237Permissions: 1238.Sy Read/Write 1239.Ed 1240.Pp 1241The 1242.Sy MAC_PROP_EN_40GFDX_CAP 1243property describes whether or not 40 Gbit/s full-duplex support is 1244enabled. 1245.It Sy MAC_PROP_ADV_100GFDX_CAP 1246.Bd -filled -compact 1247Type: 1248.Sy uint8_t | 1249Permissions: 1250.Sy Read-Only 1251.Ed 1252.Pp 1253The 1254.Sy MAC_PROP_ADV_100GFDX_CAP 1255property describes whether or not 100 Gbit/s full-duplex support is 1256advertised. 1257.It Sy MAC_PROP_EN_100GFDX_CAP 1258.Bd -filled -compact 1259Type: 1260.Sy uint8_t | 1261Permissions: 1262.Sy Read/Write 1263.Ed 1264.Pp 1265The 1266.Sy MAC_PROP_EN_100GFDX_CAP 1267property describes whether or not 100 Gbit/s full-duplex support is 1268enabled. 1269.El 1270.Ss Private Properties 1271In addition to the defined properties above, drivers are allowed to 1272define private properties. 1273These private properties are device-specific properties. 1274All private properties share the same constant, 1275.Sy MAC_PROP_PRIVATE . 1276Properties are distinguished by a name, which is a character string. 1277The list of such private properties is defined when registering with mac in the 1278.Sy m_priv_props 1279member of the 1280.Xr mac_register 9S 1281structure. 1282.Pp 1283The driver may define whatever semantics it wants for these private 1284properties. 1285They will not be listed when running 1286.Xr dladm 1M , 1287unless explicitly requested by name. 1288All such properties should start with a leading underscore character and then 1289consist of alphanumeric ASCII characters and additional underscores or hyphens. 1290.Pp 1291Properties of type 1292.Sy MAC_PROP_PRIVATE 1293may show up in all three property related entry points: 1294.Xr mc_propinfo 9E , 1295.Xr mc_getprop 9E , 1296and 1297.Xr mc_setprop 9E . 1298Device drivers should tell the different properties apart by using the 1299.Xr strcmp 9F 1300function to compare it to the set of properties that it knows about. 1301When encountering properties that it doesn't know, it should treat them 1302like all other unknown properties. 1303.Sh STATISTICS 1304The MAC framework defines a couple different sets of statistics which 1305are based on various standards for devices to implement. 1306Statistics are retrieved through the 1307.Xr mc_getstat 9E 1308entry point. 1309There are both statistics that are required for all devices and then there is a 1310separate set of Ethernet specific statistics. 1311Not all devices will support every statistic. 1312In many cases, several device registers will need to be combined to create the 1313proper stat. 1314.Pp 1315In general, if the device is not keeping track of these statistics, then 1316it is recommended that the driver store these values as a 1317.Sy uint64_t 1318to ensure that overflow does not occur. 1319.Pp 1320If a device does not support a specific statistic, then it is fine to 1321return that it is not supported. 1322The same should be used for unrecognized statistics. 1323See 1324.Xr mc_getstat 9E 1325for more information on the proper way to handle these. 1326.Ss General Device Statistics 1327The following statistics are based on MIB-II statistics from both RFC 13281213 and RFC 1573. 1329.Bl -tag -width Ds 1330.It Sy MAC_STAT_IFSPEED 1331The device's current speed in bits per second. 1332.It Sy MAC_STAT_MULTIRCV 1333The total number of received multicast packets. 1334.It Sy MAC_STAT_BRDCSTRCV 1335The total number of received broadcast packets. 1336.It Sy MAC_STAT_MULTIXMT 1337The total number of transmitted multicast packets. 1338.It Sy MAC_STAT_BRDCSTXMT 1339The total number of received broadcast packets. 1340.It Sy MAC_STAT_NORCVBUF 1341The total number of packets discarded by the hardware due to a lack of 1342receive buffers. 1343.It Sy MAC_STAT_IERRORS 1344The total number of errors detected on input. 1345.It Sy MAC_STAT_UNKNOWNS 1346The total number of received packets that were discarded because they 1347were of an unknown protocol. 1348.It Sy MAC_STAT_NOXMTBUF 1349The total number of outgoing packets dropped due to a lack of transmit 1350buffers. 1351.It Sy MAC_STAT_OERRORS 1352The total number of outgoing packets that resulted in errors. 1353.It Sy MAC_STAT_COLLISIONS 1354Total number of collisions encountered by the transmitter. 1355.It Sy MAC_STAT_RBYTES 1356The total number of 1357.Sy bytes 1358received by the device, regardless of packet type. 1359.It Sy MAC_STAT_IPACKETS 1360The total number of 1361.Sy packets 1362received by the device, regardless of packet type. 1363.It Sy MAC_STAT_OBYTES 1364The total number of 1365.Sy bytes 1366transmitted by the device, regardless of packet type. 1367.It Sy MAC_STAT_OPACKETS 1368The total number of 1369.Sy packets 1370sent by the device, regardless of packet type. 1371.It Sy MAC_STAT_UNDERFLOWS 1372The total number of packets that were smaller than the minimum sized 1373packet for the device and were therefore dropped. 1374.It Sy MAC_STAT_OVERFLOWS 1375The total number of packets that were larger than the maximum sized 1376packet for the device and were therefore dropped. 1377.El 1378.Ss Ethernet Specific Statistics 1379The following statistics are specific to Ethernet devices. 1380They refer to values from RFC 1643 and include various MII/GMII specific stats. 1381Many of these are also defined in IEEE 802.3. 1382.Bl -tag -width Ds 1383.It Sy ETHER_STAT_ADV_CAP_1000FDX 1384Indicates that the device is advertising support for 1 Gbit/s 1385full-duplex operation. 1386.It Sy ETHER_STAT_ADV_CAP_1000HDX 1387Indicates that the device is advertising support for 1 Gbit/s 1388half-duplex operation. 1389.It Sy ETHER_STAT_ADV_CAP_100FDX 1390Indicates that the device is advertising support for 100 Mbit/s 1391full-duplex operation. 1392.It Sy ETHER_STAT_ADV_CAP_100GFDX 1393Indicates that the device is advertising support for 100 Gbit/s 1394full-duplex operation. 1395.It Sy ETHER_STAT_ADV_CAP_100HDX 1396Indicates that the device is advertising support for 100 Mbit/s 1397half-duplex operation. 1398.It Sy ETHER_STAT_ADV_CAP_100T4 1399Indicates that the device is advertising support for 100 Mbit/s 1400100BASE-T4 operation. 1401.It Sy ETHER_STAT_ADV_CAP_10FDX 1402Indicates that the device is advertising support for 10 Mbit/s 1403full-duplex operation. 1404.It Sy ETHER_STAT_ADV_CAP_10GFDX 1405Indicates that the device is advertising support for 10 Gbit/s 1406full-duplex operation. 1407.It Sy ETHER_STAT_ADV_CAP_10HDX 1408Indicates that the device is advertising support for 10 Mbit/s 1409half-duplex operation. 1410.It Sy ETHER_STAT_ADV_CAP_2500FDX 1411Indicates that the device is advertising support for 2.5 Gbit/s 1412full-duplex operation. 1413.It Sy ETHER_STAT_ADV_CAP_40GFDX 1414Indicates that the device is advertising support for 40 Gbit/s 1415full-duplex operation. 1416.It Sy ETHER_STAT_ADV_CAP_5000FDX 1417Indicates that the device is advertising support for 5.0 Gbit/s 1418full-duplex operation. 1419.It Sy ETHER_STAT_ADV_CAP_ASMPAUSE 1420Indicates that the device is advertising support for receiving pause 1421frames. 1422.It Sy ETHER_STAT_ADV_CAP_AUTONEG 1423Indicates that the device is advertising support for auto-negotiation. 1424.It Sy ETHER_STAT_ADV_CAP_PAUSE 1425Indicates that the device is advertising support for generating pause 1426frames. 1427.It Sy ETHER_STAT_ADV_REMFAULT 1428Indicates that the device is advertising support for detecting faults in 1429the remote link peer. 1430.It Sy ETHER_STAT_ALIGN_ERRORS 1431Indicates the number of times an alignment error was generated by the 1432Ethernet device. 1433This is a count of packets that were not an integral number of octets and failed 1434the FCS check. 1435.It Sy ETHER_STAT_CAP_1000FDX 1436Indicates the device supports 1 Gbit/s full-duplex operation. 1437.It Sy ETHER_STAT_CAP_1000HDX 1438Indicates the device supports 1 Gbit/s half-duplex operation. 1439.It Sy ETHER_STAT_CAP_100FDX 1440Indicates the device supports 100 Mbit/s full-duplex operation. 1441.It Sy ETHER_STAT_CAP_100GFDX 1442Indicates the device supports 100 Gbit/s full-duplex operation. 1443.It Sy ETHER_STAT_CAP_100HDX 1444Indicates the device supports 100 Mbit/s half-duplex operation. 1445.It Sy ETHER_STAT_CAP_100T4 1446Indicates the device supports 100 Mbit/s 100BASE-T4 operation. 1447.It Sy ETHER_STAT_CAP_10FDX 1448Indicates the device supports 10 Mbit/s full-duplex operation. 1449.It Sy ETHER_STAT_CAP_10GFDX 1450Indicates the device supports 10 Gbit/s full-duplex operation. 1451.It Sy ETHER_STAT_CAP_10HDX 1452Indicates the device supports 10 Mbit/s half-duplex operation. 1453.It Sy ETHER_STAT_CAP_2500FDX 1454Indicates the device supports 2.5 Gbit/s full-duplex operation. 1455.It Sy ETHER_STAT_CAP_40GFDX 1456Indicates the device supports 40 Gbit/s full-duplex operation. 1457.It Sy ETHER_STAT_CAP_5000FDX 1458Indicates the device supports 5.0 Gbit/s full-duplex operation. 1459.It Sy ETHER_STAT_CAP_ASMPAUSE 1460Indicates that the device supports the ability to receive pause frames. 1461.It Sy ETHER_STAT_CAP_AUTONEG 1462Indicates that the device supports the ability to perform link 1463auto-negotiation. 1464.It Sy ETHER_STAT_CAP_PAUSE 1465Indicates that the device supports the ability to transmit pause frames. 1466.It Sy ETHER_STAT_CAP_REMFAULT 1467Indicates that the device supports the ability of detecting a remote 1468fault in a link peer. 1469.It Sy ETHER_STAT_CARRIER_ERRORS 1470Indicates the number of times that the Ethernet carrier sense condition 1471was lost or not asserted. 1472.It Sy ETHER_STAT_DEFER_XMTS 1473Indicates the number of frames for which the device was unable to 1474transmit the frame due to being busy and had to try again. 1475.It Sy ETHER_STAT_EX_COLLISIONS 1476Indicates the number of frames that failed to send due to an excessive 1477number of collisions. 1478.It Sy ETHER_STAT_FCS_ERRORS 1479Indicates the number of times that a frame check sequence failed. 1480.It Sy ETHER_STAT_FIRST_COLLISIONS 1481Indicates the number of times that a frame was eventually transmitted 1482successfully, but only after a single collision. 1483.It Sy ETHER_STAT_JABBER_ERRORS 1484Indicates the number of frames that were received that were both larger 1485than the maximum packet size and failed the frame check sequence. 1486.It Sy ETHER_STAT_LINK_ASMPAUSE 1487Indicates whether the link is currently configured to accept pause 1488frames. 1489.It Sy ETHER_STAT_LINK_AUTONEG 1490Indicates whether the current link state is a result of 1491auto-negotiation. 1492.It Sy ETHER_STAT_LINK_DUPLEX 1493Indicates the current duplex state of the link. 1494The values used here should be the same as documented for 1495.Sy MAC_PROP_DUPLEX . 1496.It Sy ETHER_STAT_LINK_PAUSE 1497Indicates whether the link is currently configured to generate pause 1498frames. 1499.It Sy ETHER_STAT_LP_CAP_1000FDX 1500Indicates the remote device supports 1 Gbit/s full-duplex operation. 1501.It Sy ETHER_STAT_LP_CAP_1000HDX 1502Indicates the remote device supports 1 Gbit/s half-duplex operation. 1503.It Sy ETHER_STAT_LP_CAP_100FDX 1504Indicates the remote device supports 100 Mbit/s full-duplex operation. 1505.It Sy ETHER_STAT_LP_CAP_100GFDX 1506Indicates the remote device supports 100 Gbit/s full-duplex operation. 1507.It Sy ETHER_STAT_LP_CAP_100HDX 1508Indicates the remote device supports 100 Mbit/s half-duplex operation. 1509.It Sy ETHER_STAT_LP_CAP_100T4 1510Indicates the remote device supports 100 Mbit/s 100BASE-T4 operation. 1511.It Sy ETHER_STAT_LP_CAP_10FDX 1512Indicates the remote device supports 10 Mbit/s full-duplex operation. 1513.It Sy ETHER_STAT_LP_CAP_10GFDX 1514Indicates the remote device supports 10 Gbit/s full-duplex operation. 1515.It Sy ETHER_STAT_LP_CAP_10HDX 1516Indicates the remote device supports 10 Mbit/s half-duplex operation. 1517.It Sy ETHER_STAT_LP_CAP_2500FDX 1518Indicates the remote device supports 2.5 Gbit/s full-duplex operation. 1519.It Sy ETHER_STAT_LP_CAP_40GFDX 1520Indicates the remote device supports 40 Gbit/s full-duplex operation. 1521.It Sy ETHER_STAT_LP_CAP_5000FDX 1522Indicates the remote device supports 5.0 Gbit/s full-duplex operation. 1523.It Sy ETHER_STAT_LP_CAP_ASMPAUSE 1524Indicates that the remote device supports the ability to receive pause 1525frames. 1526.It Sy ETHER_STAT_LP_CAP_AUTONEG 1527Indicates that the remote device supports the ability to perform link 1528auto-negotiation. 1529.It Sy ETHER_STAT_LP_CAP_PAUSE 1530Indicates that the remote device supports the ability to transmit pause 1531frames. 1532.It Sy ETHER_STAT_LP_CAP_REMFAULT 1533Indicates that the remote device supports the ability of detecting a 1534remote fault in a link peer. 1535.It Sy ETHER_STAT_MACRCV_ERRORS 1536Indicates the number of times that the internal MAC layer encountered an 1537error when attempting to receive and process a frame. 1538.It Sy ETHER_STAT_MACXMT_ERRORS 1539Indicates the number of times that the internal MAC layer encountered an 1540error when attempting to process and transmit a frame. 1541.It Sy ETHER_STAT_MULTI_COLLISIONS 1542Indicates the number of times that a frame was eventually transmitted 1543successfully, but only after more than one collision. 1544.It Sy ETHER_STAT_SQE_ERRORS 1545Indicates the number of times that an SQE error occurred. 1546The specific conditions for this error are documented in IEEE 802.3. 1547.It Sy ETHER_STAT_TOOLONG_ERRORS 1548Indicates the number of frames that were received that were longer than 1549the maximum frame size supported by the device. 1550.It Sy ETHER_STAT_TOOSHORT_ERRORS 1551Indicates the number of frames that were received that were shorter than 1552the minimum frame size supported by the device. 1553.It Sy ETHER_STAT_TX_LATE_COLLISIONS 1554Indicates the number of times a collision was detected late on the 1555device. 1556.It Sy ETHER_STAT_XCVR_ADDR 1557Indicates the address of the MII/GMII receiver address. 1558.It Sy ETHER_STAT_XCVR_ID 1559Indicates the id of the MII/GMII receiver address. 1560.It Sy ETHER_STAT_XCVR_INUSE 1561Indicates what kind of receiver is in use. 1562The following values may be used: 1563.Bl -tag -width Ds 1564.It Sy XCVR_UNDEFINED 1565The receiver type is undefined by the hardware. 1566.It Sy XCVR_NONE 1567There is no receiver in use by the hardware. 1568.It Sy XCVR_10 1569The receiver supports 10BASE-T operation. 1570.It Sy XCVR_100T4 1571The receiver supports 100BASE-T4 operation. 1572.It Sy XCVR_100X 1573The receiver supports 100BASE-TX operation. 1574.It Sy XCVR_100T2 1575The receiver supports 100BASE-T2 operation. 1576.It Sy XCVR_1000X 1577The receiver supports 1000BASE-X operation. 1578This is used for all fiber receivers. 1579.It Sy XCVR_1000T 1580The receiver supports 1000BASE-T operation. 1581This is used for all copper receivers. 1582.El 1583.El 1584.Ss Device Specific kstats 1585In addition to the defined statistics above, if the device driver 1586maintains additional statistics or the device provides additional 1587statistics, it should create its own kstats through the 1588.Xr kstat_create 9F 1589function to allow operators to observe them. 1590.Sh RECEIVE DESCRIPTOR LAYOUT 1591One of the important things that a device driver must do is lay out DMA 1592memory, generally in a ring of descriptors, into which received Ethernet 1593frames will be placed. 1594When performing this, there are a few things that drivers should 1595generally do: 1596.Bl -enum -offset indent 1597.It 1598Drivers should lay out memory so that the IP header will be 4-byte 1599aligned. 1600The IP stack expects that the beginning of an IP header will be at a 16014-byte aligned address; however, a DMA allocation will be at a 4- 1602or 8-byte aligned address by default. 1603The IP hearder is at a 14 byte offset from the beginning of the Ethernet 1604frame, leaving the IP header at a 2-byte alignment if the Ethernet frame 1605starts at the beginning of the DMA buffer. 1606If VLAN tagging is in place, then each VLAN tag adds 4 bytes, which 1607doesn't change the alignment the IP header is found at. 1608.Pp 1609As a solution to this, the driver should program the device to start 1610placing the received Ethernet frame at two bytes off of the start of the 1611DMA buffer. 1612This will make sure that no matter whether or not VLAN tags are present, 1613that the IP header will be 4-byte aligned. 1614.It 1615Drivers should try to allocate the DMA memory used for receiving frames 1616as a continuous buffer. 1617If for some reason that would not be possible, the driver should try to 1618ensure that there is enough space for all of the initial Ethernet and 1619any possible layer three and layer four headers 1620.Pq such as IP, TCP, or UDP 1621in the initial descriptor. 1622.It 1623As discussed in the 1624.Sx MBLKS AND DMA 1625section, there are multiple strategies for managing the relationship 1626between DMA data, receive descriptors, and the operating system 1627representation of a packet in the 1628.Xr mblk 9S 1629structure. 1630Drivers must limit their resource consumption. 1631See the 1632.Sy Considerations 1633section of 1634.Sx MBLKS AND DMA 1635for more on this. 1636.El 1637.Sh TX STALL DETECTION, DEVICE RESETS, AND FAULT MANAGEMENT 1638Device drivers are the first line of defense for dealing with broken 1639devices and bugs in their firmware. 1640While most devices will rarely fail, it is important that when designing and 1641implementing the device driver that particular attention is paid in the design 1642with respect to RAS (Reliability, Availability, and Serviceability). 1643While everything described in this section is optional, it is highly recommended 1644that all new device drivers follow these guidelines. 1645.Pp 1646The Fault Management Architecture (FMA) provides facilities for 1647detecting and reporting various classes of defects and faults. 1648Specifically for networking device drivers, issues that should be 1649detected and reported include: 1650.Bl -bullet -offset indent 1651.It 1652Device internal uncorrectable errors 1653.It 1654Device internal correctable errors 1655.It 1656PCI and PCI Express transport errors 1657.It 1658Device temperature alarms 1659.It 1660Device transmission stalls 1661.It 1662Device communication timeouts 1663.It 1664High invalid interrupts 1665.El 1666.Pp 1667All such errors fall into three primary categories: 1668.Bl -enum -offset indent 1669.It 1670Errors detected by the Fault Management Architecture 1671.It 1672Errors detected by the device and indicated to the device driver 1673.It 1674Errors detected by the device driver 1675.El 1676.Ss Fault Management Setup and Teardown 1677Drivers should initialize support for the fault management framework by 1678calling 1679.Xr ddi_fm_init 9F 1680from their 1681.Xr attach 9E 1682routine. 1683By registering with the fault management framework, a device driver is given the 1684chance to detect and notice transport errors as well as report other errors that 1685exist. 1686While a device driver does not need to indicate that it is capable of all such 1687capabilities described in 1688.Xr ddi_fm_init 9F , 1689we suggest that device drivers at least register the 1690.Sy DDI_FM_EREPORT_CAPABLE 1691so as to allow the driver to report issues that it detects. 1692.Pp 1693If the driver registers with the fault management framework during its 1694.Xr attach 9E 1695entry point, it must call 1696.Xr ddi_fm_fini 9F 1697during its 1698.Xr detach 9E 1699entry point. 1700.Ss Transport Errors 1701Many modern networking devices leverage PCI or PCI Express. 1702As such, there are two primary ways that device drivers access data: they either 1703memory map device registers and use routines like 1704.Xr ddi_get8 9F 1705and 1706.Xr ddi_put8 9F 1707or they use direct memory access (DMA). 1708New device drivers should always enable checking of the transport layer by 1709marking their support in the 1710.Xr ddi_device_acc_attr 9S 1711structure and using routines like 1712.Xr ddi_fm_acc_err_get 9F 1713and 1714.Xr ddi_fm_dma_err_get 9F 1715to detect if errors have occurred. 1716.Ss Device Indicated Errors 1717Many devices have capabilities to announce to a device driver that a 1718fatal correctable error or uncorrectable error has occurred. 1719Other devices have the ability to indicate that various physical issues have 1720occurred such as a fan failing or a temperature sensor having fired. 1721.Pp 1722Drivers should wire themselves to receive notifications when these 1723events occur. 1724The means and capabilities will vary from device to device. 1725For example, some devices will generate information about these notifications 1726through special interrupts. 1727Other devices may have a register that software can poll. 1728In the cases where polling is required, driver writers should try not to poll 1729too frequently and should generally only poll when the device is actively being 1730used, e.g. between calls to the 1731.Xr mc_start 9E 1732and 1733.Xr mc_stop 9E 1734entry points. 1735.Ss Driver Transmit Stall Detection 1736One of the primary responsibilities of a hardened device driver is to 1737perform transmit stall detection. 1738The core idea behind tx stall detection is that the driver should record when 1739it's getting activity related to when data has been successfully transmitted. 1740Most devices should be transmitting data on a regular basis as long as the link 1741is up. 1742If it is not, then this may indicate that the device is stuck and needs to be 1743reset. 1744At this time, the MAC framework does not provide any resources for performing 1745these checks; however, polling on each individual transmit ring for the last 1746completion time while something is actively being transmitted through the use of 1747routines such as 1748.Xr timeout 9F 1749may be a reasonable starting point. 1750.Ss Driver Command Timeout Detection 1751Each device is programmed in different ways. 1752Some devices are programmed through asynchronous commands while others are 1753programmed by writing directly to memory mapped registers. 1754If a device receives asynchronous replies to commands, then the device driver 1755should set reasonable timeouts for all such commands and plan on detecting them. 1756If a timeout occurs, the driver should presume that there is an issue with the 1757hardware and proceed to abort the command or reset the device. 1758.Pp 1759Many devices do not have such a communication mechanism. 1760However, whenever there is some activity where the device driver must wait, then 1761it should be prepared for the fact that the device may never get back to 1762it and react appropriately by performing some kind of device reset. 1763.Ss Reacting to Errors 1764When any of the above categories of errors has been triggered, the 1765behavior that the device driver should take depends on the kind of 1766error. 1767If a fatal error, for example, a transport error, a transmit stall was detected, 1768or the device indicated an uncorrectable error was detected, then it is 1769important that the driver take the following steps: 1770.Bl -enum -offset indent 1771.It 1772Set a flag in the device driver's state that indicates that it has hit 1773an error condition. 1774When this error condition flag is asserted, transmitted packets should be 1775accepted and dropped and actions that would require writing to the device state 1776should fail with an error. 1777This flag should remain until the device has been successfully restarted. 1778.It 1779If the error was not a transport error that was indicated by the fault 1780management architecture, e.g. a transport error that was detected, then 1781the device driver should post an 1782.Sy ereport 1783indicating what has occurred with the 1784.Xr ddi_fm_ereport_post 9F 1785function. 1786.It 1787The device driver should indicate that the device's service was lost 1788with a call to 1789.Xr ddi_fm_service_impact 9F 1790using the symbol 1791.Sy DDI_SERVICE_LOST . 1792.It 1793At this point the device driver should issue a device reset through some 1794device-specific means. 1795.It 1796When the device reset has been completed, then the device driver should 1797restore all of the programmed state to the device. 1798This includes things like the current MTU, advertised auto-negotiation speeds, 1799MAC address filters, and more. 1800.It 1801Finally, when service has been restored, the device driver should call 1802.Xr ddi_fm_service_impact 9F 1803using the symbol 1804.Sy DDI_SERVICE_RESTORED . 1805.El 1806.Pp 1807When a non-fatal error occurs, then the device driver should submit an 1808ereport and should optionally mark the device degraded using 1809.Xr ddi_fm_service_impact 9F 1810with the 1811.Sy DDI_SERVICE_DEGRADED 1812value depending on the nature of the problem that has occurred. 1813.Pp 1814Device drivers should never make the decision to remove a device from 1815service based on errors that have occurred nor should they panic the 1816system. 1817Rather, the device driver should always try to notify the operating system with 1818various ereports and allow its policy decisions to occur. 1819The decision to retire a device lies in the hands of the fault management 1820architecture. 1821It knows more about the operator's intent and the surrounding system's state 1822than the device driver itself does and it will make the call to offline and 1823retire the device if it is required. 1824.Ss Device Resets 1825When resetting a device, a device driver must exercise caution. 1826If a device driver has not been written to plan for a device reset, then it 1827may not correctly restore the device's state after such a reset. 1828Such state should be stored in the instance's private state data as the MAC 1829framework does not know about device resets and will not inform the 1830device again about the expected, programmed state. 1831.Pp 1832One wrinkle with device resets is that many networking cards show up as 1833multiple PCI functions on a single device, for example, each port may 1834show up as a separate function and thus have a separate instance of the 1835device driver attached. 1836When resetting a function, device driver writers should carefully read the 1837device programming manuals and verify whether or not a reset impacts only the 1838stalled function or if it impacts all function across the device. 1839.Pp 1840If the only way to reset a given function is through the device, then 1841this may require more coordination and work on the part of the device 1842driver to ensure that all the other instances are correctly restored. 1843In cases where this occurs, some devices offer ways of injecting 1844interrupts onto those other functions to notify them that this is 1845occurring. 1846.Sh MBLKS AND DMA 1847The networking stack manages framed data through the use of the 1848.Xr mblk 9S 1849structure. 1850The mblk allows for a single message to be made up of individual blocks. 1851Each part is linked together through its 1852.Sy b_cont 1853member. 1854However, it also allows for multiple messages to be chained together through the 1855use of the 1856.Sy b_next 1857member. 1858While the networking stack works with these structures, device drivers generally 1859work with DMA regions. 1860There are two different strategies that device drivers use for handling these 1861two different cases: copying and binding. 1862.Ss Copying Data 1863The first way that device drivers handle interfacing between the two is 1864by having two separate regions of memory. 1865One part is memory which has been allocated for DMA through a call to 1866.Xr ddi_dma_mem_alloc 9F 1867and the other is memory associated with the memory block. 1868.Pp 1869In this case, a driver will use 1870.Xr bcopy 9F 1871to copy memory between the two distinct regions. 1872When transmitting a packet, it will copy the memory from the mblk_t to the DMA 1873region. 1874When receiving memory, it will allocate a mblk_t through the 1875.Xr allocb 9F 1876routine, copy the memory across with 1877.Xr bcopy 9F , 1878and then increment the mblk_t's 1879.Sy w_ptr 1880structure. 1881.Pp 1882If, when receiving, memory is not available for a new message block, 1883then the frame should be skipped and effectively dropped. 1884A kstat should be bumped when such an occasion occurs. 1885.Ss Binding Data 1886An alternative approach to copying data is to use DMA binding. 1887When using DMA binding, the OS takes care of mapping between DMA memory and 1888normal device memory. 1889The exact process is a bit different between transmit and receive. 1890.Pp 1891When transmitting a device driver has an mblk_t and needs to call the 1892.Xr ddi_dma_addr_bind_handle 9F 1893function to bind it to an already existing DMA handle. 1894At that point, it will receive various DMA cookies that it can use to obtain the 1895addresses to program the device with for transmitting data. 1896Once the transmit is done, the driver must then make sure to call 1897.Xr freemsg 9F 1898to release the data. 1899It must not call 1900.Xr freemsg 9F 1901before it receives an interrupt from the device indicating that the data 1902has been transmitted, otherwise it risks sending arbitrary kernel 1903memory. 1904.Pp 1905When receiving data, the device can perform a similar operation. 1906First, it must bind the DMA memory into the kernel's virtual memory address 1907space through a call to the 1908.Xr ddi_dma_addr_bind_handle 9F 1909function if it has not already. 1910Once it has, it must then call 1911.Xr desballoc 9F 1912to try and create a new mblk_t which leverages the associated memory. 1913It can then pass that mblk_t up to the stack. 1914.Ss Considerations 1915When deciding which of these options to use, there are many different 1916considerations that must be made. 1917The answer as to whether to bind memory or to copy data is not always simpler. 1918.Pp 1919The first thing to remember is that DMA resources may be finite on a 1920given platform. 1921Consider the case of receiving data. 1922A device driver that binds one of its receive descriptors may not get it back 1923for quite some time as it may be used by the kernel until an application 1924actually consumes it. 1925Device drivers that try to bind memory for receive, often work with the 1926constraint that they must be able to replace that DMA memory with another DMA 1927descriptor. 1928If they were not replaced, then eventually the device would not be able to 1929receive additional data into the ring. 1930.Pp 1931On the other hand, particularly for larger frames, copying every packet 1932from one buffer to another can be a source of additional latency and 1933memory waste in the system. 1934For larger copies, the cost of copying may dwarf any potential cost of 1935performing DMA binding. 1936.Pp 1937For device driver authors that are unsure of what to do, they should 1938first employ the copying method to simplify the act of writing the 1939device driver. 1940The copying method is simpler and also allows the device driver author not to 1941worry about allocated DMA memory that is still outstanding when it is asked to 1942unload. 1943.Pp 1944If device driver writers are worried about the cost, it is recommended 1945to make the decision as to whether or not to copy or bind DMA data 1946a separate private property for both transmitting and receiving. 1947That private property should indicate the size of the received frame at which 1948to switch from one format to the other. 1949This way, data can be gathered to determine what the impact of each method is on 1950a given platform. 1951.Sh SEE ALSO 1952.Xr dladm 1M , 1953.Xr driver.conf 4 , 1954.Xr ieee802.3 5 , 1955.Xr dlpi 7P , 1956.Xr _fini 9E , 1957.Xr _info 9E , 1958.Xr _init 9E , 1959.Xr attach 9E , 1960.Xr close 9E , 1961.Xr detach 9E , 1962.Xr mc_close 9E , 1963.Xr mc_getcapab 9E , 1964.Xr mc_getprop 9E , 1965.Xr mc_getstat 9E , 1966.Xr mc_multicst 9E , 1967.Xr mc_open 9E , 1968.Xr mc_propinfo 9E , 1969.Xr mc_setpromisc 9E , 1970.Xr mc_setprop 9E , 1971.Xr mc_start 9E , 1972.Xr mc_stop 9E , 1973.Xr mc_tx 9E , 1974.Xr mc_unicst 9E , 1975.Xr open 9E , 1976.Xr allocb 9F , 1977.Xr bcopy 9F , 1978.Xr ddi_dma_addr_bind_handle 9F , 1979.Xr ddi_dma_mem_alloc 9F , 1980.Xr ddi_fm_acc_err_get 9F , 1981.Xr ddi_fm_dma_err_get 9F , 1982.Xr ddi_fm_ereport_post 9F , 1983.Xr ddi_fm_fini 9F , 1984.Xr ddi_fm_init 9F , 1985.Xr ddi_fm_service_impact 9F , 1986.Xr ddi_get8 9F , 1987.Xr ddi_put8 9F , 1988.Xr desballoc 9F , 1989.Xr freemsg 9F , 1990.Xr kstat_create 9F , 1991.Xr mac_alloc 9F , 1992.Xr mac_fini_ops 9F , 1993.Xr mac_free 9F , 1994.Xr mac_hcksum_get 9F , 1995.Xr mac_hcksum_set 9F , 1996.Xr mac_init_ops 9F , 1997.Xr mac_link_update 9F , 1998.Xr mac_lso_get 9F , 1999.Xr mac_maxsdu_update 9F , 2000.Xr mac_prop_info_set_default_link_flowctrl 9F , 2001.Xr mac_prop_info_set_default_str 9F , 2002.Xr mac_prop_info_set_default_uint32 9F , 2003.Xr mac_prop_info_set_default_uint64 9F , 2004.Xr mac_prop_info_set_default_uint8 9F , 2005.Xr mac_prop_info_set_perm 9F , 2006.Xr mac_prop_info_set_range_uint32 9F , 2007.Xr mac_register 9F , 2008.Xr mac_rx 9F , 2009.Xr mac_unregister 9F , 2010.Xr mod_install 9F , 2011.Xr mod_remove 9F , 2012.Xr strcmp 9F , 2013.Xr timeout 9F , 2014.Xr cb_ops 9S , 2015.Xr ddi_device_acc_attr 9S , 2016.Xr dev_ops 9S , 2017.Xr mac_callbacks 9S , 2018.Xr mac_register 9S , 2019.Xr mblk 9S , 2020.Xr modldrv 9S , 2021.Xr modlinkage 9S 2022.Rs 2023.%A McCloghrie, K. 2024.%A Rose, M. 2025.%T RFC 1213 Management Information Base for Network Management of 2026.%T TCP/IP-based internets: MIB-II 2027.%D March 1991 2028.Re 2029.Rs 2030.%A McCloghrie, K. 2031.%A Kastenholz, F. 2032.%T RFC 1573 Evolution of the Interfaces Group of MIB-II 2033.%D January 1994 2034.Re 2035.Rs 2036.%A Kastenholz, F. 2037.%T RFC 1643 Definitions of Managed Objects for the Ethernet-like 2038.%T Interface Types 2039.Re 2040