1.. SPDX-License-Identifier: GPL-2.0 2 3=================================== 4Linux Ethernet Bonding Driver HOWTO 5=================================== 6 7Latest update: 27 April 2011 8 9Initial release: Thomas Davis <tadavis at lbl.gov> 10 11Corrections, HA extensions: 2000/10/03-15: 12 13 - Willy Tarreau <willy at meta-x.org> 14 - Constantine Gavrilov <const-g at xpert.com> 15 - Chad N. Tindel <ctindel at ieee dot org> 16 - Janice Girouard <girouard at us dot ibm dot com> 17 - Jay Vosburgh <fubar at us dot ibm dot com> 18 19Reorganized and updated Feb 2005 by Jay Vosburgh 20Added Sysfs information: 2006/04/24 21 22 - Mitch Williams <mitch.a.williams at intel.com> 23 24Introduction 25============ 26 27The Linux bonding driver provides a method for aggregating 28multiple network interfaces into a single logical "bonded" interface. 29The behavior of the bonded interfaces depends upon the mode; generally 30speaking, modes provide either hot standby or load balancing services. 31Additionally, link integrity monitoring may be performed. 32 33The bonding driver originally came from Donald Becker's 34beowulf patches for kernel 2.0. It has changed quite a bit since, and 35the original tools from extreme-linux and beowulf sites will not work 36with this version of the driver. 37 38For new versions of the driver, updated userspace tools, and 39who to ask for help, please follow the links at the end of this file. 40 41.. Table of Contents 42 43 1. Bonding Driver Installation 44 45 2. Bonding Driver Options 46 47 3. Configuring Bonding Devices 48 3.1 Configuration with Sysconfig Support 49 3.1.1 Using DHCP with Sysconfig 50 3.1.2 Configuring Multiple Bonds with Sysconfig 51 3.2 Configuration with Initscripts Support 52 3.2.1 Using DHCP with Initscripts 53 3.2.2 Configuring Multiple Bonds with Initscripts 54 3.3 Configuring Bonding Manually with Ifenslave 55 3.3.1 Configuring Multiple Bonds Manually 56 3.4 Configuring Bonding Manually via Sysfs 57 3.5 Configuration with Interfaces Support 58 3.6 Overriding Configuration for Special Cases 59 3.7 Configuring LACP for 802.3ad mode in a more secure way 60 61 4. Querying Bonding Configuration 62 4.1 Bonding Configuration 63 4.2 Network Configuration 64 65 5. Switch Configuration 66 67 6. 802.1q VLAN Support 68 69 7. Link Monitoring 70 7.1 ARP Monitor Operation 71 7.2 Configuring Multiple ARP Targets 72 7.3 MII Monitor Operation 73 74 8. Potential Trouble Sources 75 8.1 Adventures in Routing 76 8.2 Ethernet Device Renaming 77 8.3 Painfully Slow Or No Failed Link Detection By Miimon 78 79 9. SNMP agents 80 81 10. Promiscuous mode 82 83 11. Configuring Bonding for High Availability 84 11.1 High Availability in a Single Switch Topology 85 11.2 High Availability in a Multiple Switch Topology 86 11.2.1 HA Bonding Mode Selection for Multiple Switch Topology 87 11.2.2 HA Link Monitoring for Multiple Switch Topology 88 89 12. Configuring Bonding for Maximum Throughput 90 12.1 Maximum Throughput in a Single Switch Topology 91 12.1.1 MT Bonding Mode Selection for Single Switch Topology 92 12.1.2 MT Link Monitoring for Single Switch Topology 93 12.2 Maximum Throughput in a Multiple Switch Topology 94 12.2.1 MT Bonding Mode Selection for Multiple Switch Topology 95 12.2.2 MT Link Monitoring for Multiple Switch Topology 96 97 13. Switch Behavior Issues 98 13.1 Link Establishment and Failover Delays 99 13.2 Duplicated Incoming Packets 100 101 14. Hardware Specific Considerations 102 14.1 IBM BladeCenter 103 104 15. Frequently Asked Questions 105 106 16. Resources and Links 107 108 1091. Bonding Driver Installation 110============================== 111 112Most popular distro kernels ship with the bonding driver 113already available as a module. If your distro does not, or you 114have need to compile bonding from source (e.g., configuring and 115installing a mainline kernel from kernel.org), you'll need to perform 116the following steps: 117 1181.1 Configure and build the kernel with bonding 119----------------------------------------------- 120 121The current version of the bonding driver is available in the 122drivers/net/bonding subdirectory of the most recent kernel source 123(which is available on http://kernel.org). Most users "rolling their 124own" will want to use the most recent kernel from kernel.org. 125 126Configure kernel with "make menuconfig" (or "make xconfig" or 127"make config"), then select "Bonding driver support" in the "Network 128device support" section. It is recommended that you configure the 129driver as module since it is currently the only way to pass parameters 130to the driver or configure more than one bonding device. 131 132Build and install the new kernel and modules. 133 1341.2 Bonding Control Utility 135--------------------------- 136 137It is recommended to configure bonding via iproute2 (netlink) 138or sysfs, the old ifenslave control utility is obsolete. 139 1402. Bonding Driver Options 141========================= 142 143Options for the bonding driver are supplied as parameters to the 144bonding module at load time, or are specified via sysfs. 145 146Module options may be given as command line arguments to the 147insmod or modprobe command, but are usually specified in either the 148``/etc/modprobe.d/*.conf`` configuration files, or in a distro-specific 149configuration file (some of which are detailed in the next section). 150 151Details on bonding support for sysfs is provided in the 152"Configuring Bonding Manually via Sysfs" section, below. 153 154The available bonding driver parameters are listed below. If a 155parameter is not specified the default value is used. When initially 156configuring a bond, it is recommended "tail -f /var/log/messages" be 157run in a separate window to watch for bonding driver error messages. 158 159It is critical that either the miimon or arp_interval and 160arp_ip_target parameters be specified, otherwise serious network 161degradation will occur during link failures. Very few devices do not 162support at least miimon, so there is really no reason not to use it. 163 164Options with textual values will accept either the text name 165or, for backwards compatibility, the option value. E.g., 166"mode=802.3ad" and "mode=4" set the same mode. 167 168The parameters are as follows: 169 170active_slave 171 172 Specifies the new active slave for modes that support it 173 (active-backup, balance-alb and balance-tlb). Possible values 174 are the name of any currently enslaved interface, or an empty 175 string. If a name is given, the slave and its link must be up in order 176 to be selected as the new active slave. If an empty string is 177 specified, the current active slave is cleared, and a new active 178 slave is selected automatically. 179 180 Note that this is only available through the sysfs interface. No module 181 parameter by this name exists. 182 183 The normal value of this option is the name of the currently 184 active slave, or the empty string if there is no active slave or 185 the current mode does not use an active slave. 186 187ad_actor_sys_prio 188 189 In an AD system, this specifies the system priority. The allowed range 190 is 1 - 65535. If the value is not specified, it takes 65535 as the 191 default value. 192 193 This parameter has effect only in 802.3ad mode and is available through 194 SysFs interface. 195 196ad_actor_system 197 198 In an AD system, this specifies the mac-address for the actor in 199 protocol packet exchanges (LACPDUs). The value cannot be a multicast 200 address. If the all-zeroes MAC is specified, bonding will internally 201 use the MAC of the bond itself. It is preferred to have the 202 local-admin bit set for this mac but driver does not enforce it. If 203 the value is not given then system defaults to using the masters' 204 mac address as actors' system address. 205 206 This parameter has effect only in 802.3ad mode and is available through 207 SysFs interface. 208 209ad_select 210 211 Specifies the 802.3ad aggregation selection logic to use. The 212 possible values and their effects are: 213 214 stable or 0 215 216 The active aggregator is chosen by largest aggregate 217 bandwidth. 218 219 Reselection of the active aggregator occurs only when all 220 slaves of the active aggregator are down or the active 221 aggregator has no slaves. 222 223 This is the default value. 224 225 bandwidth or 1 226 227 The active aggregator is chosen by largest aggregate 228 bandwidth. Reselection occurs if: 229 230 - A slave is added to or removed from the bond 231 232 - Any slave's link state changes 233 234 - Any slave's 802.3ad association state changes 235 236 - The bond's administrative state changes to up 237 238 count or 2 239 240 The active aggregator is chosen by the largest number of 241 ports (slaves). Reselection occurs as described under the 242 "bandwidth" setting, above. 243 244 The bandwidth and count selection policies permit failover of 245 802.3ad aggregations when partial failure of the active aggregator 246 occurs. This keeps the aggregator with the highest availability 247 (either in bandwidth or in number of ports) active at all times. 248 249 This option was added in bonding version 3.4.0. 250 251ad_user_port_key 252 253 In an AD system, the port-key has three parts as shown below - 254 255 ===== ============ 256 Bits Use 257 ===== ============ 258 00 Duplex 259 01-05 Speed 260 06-15 User-defined 261 ===== ============ 262 263 This defines the upper 10 bits of the port key. The values can be 264 from 0 - 1023. If not given, the system defaults to 0. 265 266 This parameter has effect only in 802.3ad mode and is available through 267 SysFs interface. 268 269all_slaves_active 270 271 Specifies that duplicate frames (received on inactive ports) should be 272 dropped (0) or delivered (1). 273 274 Normally, bonding will drop duplicate frames (received on inactive 275 ports), which is desirable for most users. But there are some times 276 it is nice to allow duplicate frames to be delivered. 277 278 The default value is 0 (drop duplicate frames received on inactive 279 ports). 280 281arp_interval 282 283 Specifies the ARP link monitoring frequency in milliseconds. 284 285 The ARP monitor works by periodically checking the slave 286 devices to determine whether they have sent or received 287 traffic recently (the precise criteria depends upon the 288 bonding mode, and the state of the slave). Regular traffic is 289 generated via ARP probes issued for the addresses specified by 290 the arp_ip_target option. 291 292 This behavior can be modified by the arp_validate option, 293 below. 294 295 If ARP monitoring is used in an etherchannel compatible mode 296 (modes 0 and 2), the switch should be configured in a mode 297 that evenly distributes packets across all links. If the 298 switch is configured to distribute the packets in an XOR 299 fashion, all replies from the ARP targets will be received on 300 the same link which could cause the other team members to 301 fail. ARP monitoring should not be used in conjunction with 302 miimon. A value of 0 disables ARP monitoring. The default 303 value is 0. 304 305arp_ip_target 306 307 Specifies the IP addresses to use as ARP monitoring peers when 308 arp_interval is > 0. These are the targets of the ARP request 309 sent to determine the health of the link to the targets. 310 Specify these values in ddd.ddd.ddd.ddd format. Multiple IP 311 addresses must be separated by a comma. At least one IP 312 address must be given for ARP monitoring to function. The 313 maximum number of targets that can be specified is 16. The 314 default value is no IP addresses. 315 316ns_ip6_target 317 318 Specifies the IPv6 addresses to use as IPv6 monitoring peers when 319 arp_interval is > 0. These are the targets of the NS request 320 sent to determine the health of the link to the targets. 321 Specify these values in ffff:ffff::ffff:ffff format. Multiple IPv6 322 addresses must be separated by a comma. At least one IPv6 323 address must be given for NS/NA monitoring to function. The 324 maximum number of targets that can be specified is 16. The 325 default value is no IPv6 addresses. 326 327arp_validate 328 329 Specifies whether or not ARP probes and replies should be 330 validated in any mode that supports arp monitoring, or whether 331 non-ARP traffic should be filtered (disregarded) for link 332 monitoring purposes. 333 334 Possible values are: 335 336 none or 0 337 338 No validation or filtering is performed. 339 340 active or 1 341 342 Validation is performed only for the active slave. 343 344 backup or 2 345 346 Validation is performed only for backup slaves. 347 348 all or 3 349 350 Validation is performed for all slaves. 351 352 filter or 4 353 354 Filtering is applied to all slaves. No validation is 355 performed. 356 357 filter_active or 5 358 359 Filtering is applied to all slaves, validation is performed 360 only for the active slave. 361 362 filter_backup or 6 363 364 Filtering is applied to all slaves, validation is performed 365 only for backup slaves. 366 367 Validation: 368 369 Enabling validation causes the ARP monitor to examine the incoming 370 ARP requests and replies, and only consider a slave to be up if it 371 is receiving the appropriate ARP traffic. 372 373 For an active slave, the validation checks ARP replies to confirm 374 that they were generated by an arp_ip_target. Since backup slaves 375 do not typically receive these replies, the validation performed 376 for backup slaves is on the broadcast ARP request sent out via the 377 active slave. It is possible that some switch or network 378 configurations may result in situations wherein the backup slaves 379 do not receive the ARP requests; in such a situation, validation 380 of backup slaves must be disabled. 381 382 The validation of ARP requests on backup slaves is mainly helping 383 bonding to decide which slaves are more likely to work in case of 384 the active slave failure, it doesn't really guarantee that the 385 backup slave will work if it's selected as the next active slave. 386 387 Validation is useful in network configurations in which multiple 388 bonding hosts are concurrently issuing ARPs to one or more targets 389 beyond a common switch. Should the link between the switch and 390 target fail (but not the switch itself), the probe traffic 391 generated by the multiple bonding instances will fool the standard 392 ARP monitor into considering the links as still up. Use of 393 validation can resolve this, as the ARP monitor will only consider 394 ARP requests and replies associated with its own instance of 395 bonding. 396 397 Filtering: 398 399 Enabling filtering causes the ARP monitor to only use incoming ARP 400 packets for link availability purposes. Arriving packets that are 401 not ARPs are delivered normally, but do not count when determining 402 if a slave is available. 403 404 Filtering operates by only considering the reception of ARP 405 packets (any ARP packet, regardless of source or destination) when 406 determining if a slave has received traffic for link availability 407 purposes. 408 409 Filtering is useful in network configurations in which significant 410 levels of third party broadcast traffic would fool the standard 411 ARP monitor into considering the links as still up. Use of 412 filtering can resolve this, as only ARP traffic is considered for 413 link availability purposes. 414 415 This option was added in bonding version 3.1.0. 416 417arp_all_targets 418 419 Specifies the quantity of arp_ip_targets that must be reachable 420 in order for the ARP monitor to consider a slave as being up. 421 This option affects only active-backup mode for slaves with 422 arp_validation enabled. 423 424 Possible values are: 425 426 any or 0 427 428 consider the slave up only when any of the arp_ip_targets 429 is reachable 430 431 all or 1 432 433 consider the slave up only when all of the arp_ip_targets 434 are reachable 435 436arp_missed_max 437 438 Specifies the number of arp_interval monitor checks that must 439 fail in order for an interface to be marked down by the ARP monitor. 440 441 In order to provide orderly failover semantics, backup interfaces 442 are permitted an extra monitor check (i.e., they must fail 443 arp_missed_max + 1 times before being marked down). 444 445 The default value is 2, and the allowable range is 1 - 255. 446 447downdelay 448 449 Specifies the time, in milliseconds, to wait before disabling 450 a slave after a link failure has been detected. This option 451 is only valid for the miimon link monitor. The downdelay 452 value should be a multiple of the miimon value; if not, it 453 will be rounded down to the nearest multiple. The default 454 value is 0. 455 456fail_over_mac 457 458 Specifies whether active-backup mode should set all slaves to 459 the same MAC address at enslavement (the traditional 460 behavior), or, when enabled, perform special handling of the 461 bond's MAC address in accordance with the selected policy. 462 463 Possible values are: 464 465 none or 0 466 467 This setting disables fail_over_mac, and causes 468 bonding to set all slaves of an active-backup bond to 469 the same MAC address at enslavement time. This is the 470 default. 471 472 active or 1 473 474 The "active" fail_over_mac policy indicates that the 475 MAC address of the bond should always be the MAC 476 address of the currently active slave. The MAC 477 address of the slaves is not changed; instead, the MAC 478 address of the bond changes during a failover. 479 480 This policy is useful for devices that cannot ever 481 alter their MAC address, or for devices that refuse 482 incoming broadcasts with their own source MAC (which 483 interferes with the ARP monitor). 484 485 The down side of this policy is that every device on 486 the network must be updated via gratuitous ARP, 487 vs. just updating a switch or set of switches (which 488 often takes place for any traffic, not just ARP 489 traffic, if the switch snoops incoming traffic to 490 update its tables) for the traditional method. If the 491 gratuitous ARP is lost, communication may be 492 disrupted. 493 494 When this policy is used in conjunction with the mii 495 monitor, devices which assert link up prior to being 496 able to actually transmit and receive are particularly 497 susceptible to loss of the gratuitous ARP, and an 498 appropriate updelay setting may be required. 499 500 follow or 2 501 502 The "follow" fail_over_mac policy causes the MAC 503 address of the bond to be selected normally (normally 504 the MAC address of the first slave added to the bond). 505 However, the second and subsequent slaves are not set 506 to this MAC address while they are in a backup role; a 507 slave is programmed with the bond's MAC address at 508 failover time (and the formerly active slave receives 509 the newly active slave's MAC address). 510 511 This policy is useful for multiport devices that 512 either become confused or incur a performance penalty 513 when multiple ports are programmed with the same MAC 514 address. 515 516 517 The default policy is none, unless the first slave cannot 518 change its MAC address, in which case the active policy is 519 selected by default. 520 521 This option may be modified via sysfs only when no slaves are 522 present in the bond. 523 524 This option was added in bonding version 3.2.0. The "follow" 525 policy was added in bonding version 3.3.0. 526 527lacp_active 528 Option specifying whether to send LACPDU frames periodically. 529 530 off or 0 531 LACPDU frames acts as "speak when spoken to". 532 533 on or 1 534 LACPDU frames are sent along the configured links 535 periodically. See lacp_rate for more details. 536 537 The default is on. 538 539lacp_rate 540 541 Option specifying the rate in which we'll ask our link partner 542 to transmit LACPDU packets in 802.3ad mode. Possible values 543 are: 544 545 slow or 0 546 Request partner to transmit LACPDUs every 30 seconds 547 548 fast or 1 549 Request partner to transmit LACPDUs every 1 second 550 551 The default is slow. 552 553max_bonds 554 555 Specifies the number of bonding devices to create for this 556 instance of the bonding driver. E.g., if max_bonds is 3, and 557 the bonding driver is not already loaded, then bond0, bond1 558 and bond2 will be created. The default value is 1. Specifying 559 a value of 0 will load bonding, but will not create any devices. 560 561miimon 562 563 Specifies the MII link monitoring frequency in milliseconds. 564 This determines how often the link state of each slave is 565 inspected for link failures. A value of zero disables MII 566 link monitoring. A value of 100 is a good starting point. 567 The use_carrier option, below, affects how the link state is 568 determined. See the High Availability section for additional 569 information. The default value is 0. 570 571min_links 572 573 Specifies the minimum number of links that must be active before 574 asserting carrier. It is similar to the Cisco EtherChannel min-links 575 feature. This allows setting the minimum number of member ports that 576 must be up (link-up state) before marking the bond device as up 577 (carrier on). This is useful for situations where higher level services 578 such as clustering want to ensure a minimum number of low bandwidth 579 links are active before switchover. This option only affect 802.3ad 580 mode. 581 582 The default value is 0. This will cause carrier to be asserted (for 583 802.3ad mode) whenever there is an active aggregator, regardless of the 584 number of available links in that aggregator. Note that, because an 585 aggregator cannot be active without at least one available link, 586 setting this option to 0 or to 1 has the exact same effect. 587 588mode 589 590 Specifies one of the bonding policies. The default is 591 balance-rr (round robin). Possible values are: 592 593 balance-rr or 0 594 595 Round-robin policy: Transmit packets in sequential 596 order from the first available slave through the 597 last. This mode provides load balancing and fault 598 tolerance. 599 600 active-backup or 1 601 602 Active-backup policy: Only one slave in the bond is 603 active. A different slave becomes active if, and only 604 if, the active slave fails. The bond's MAC address is 605 externally visible on only one port (network adapter) 606 to avoid confusing the switch. 607 608 In bonding version 2.6.2 or later, when a failover 609 occurs in active-backup mode, bonding will issue one 610 or more gratuitous ARPs on the newly active slave. 611 One gratuitous ARP is issued for the bonding master 612 interface and each VLAN interfaces configured above 613 it, provided that the interface has at least one IP 614 address configured. Gratuitous ARPs issued for VLAN 615 interfaces are tagged with the appropriate VLAN id. 616 617 This mode provides fault tolerance. The primary 618 option, documented below, affects the behavior of this 619 mode. 620 621 balance-xor or 2 622 623 XOR policy: Transmit based on the selected transmit 624 hash policy. The default policy is a simple [(source 625 MAC address XOR'd with destination MAC address XOR 626 packet type ID) modulo slave count]. Alternate transmit 627 policies may be selected via the xmit_hash_policy option, 628 described below. 629 630 This mode provides load balancing and fault tolerance. 631 632 broadcast or 3 633 634 Broadcast policy: transmits everything on all slave 635 interfaces. This mode provides fault tolerance. 636 637 802.3ad or 4 638 639 IEEE 802.3ad Dynamic link aggregation. Creates 640 aggregation groups that share the same speed and 641 duplex settings. Utilizes all slaves in the active 642 aggregator according to the 802.3ad specification. 643 644 Slave selection for outgoing traffic is done according 645 to the transmit hash policy, which may be changed from 646 the default simple XOR policy via the xmit_hash_policy 647 option, documented below. Note that not all transmit 648 policies may be 802.3ad compliant, particularly in 649 regards to the packet mis-ordering requirements of 650 section 43.2.4 of the 802.3ad standard. Differing 651 peer implementations will have varying tolerances for 652 noncompliance. 653 654 Prerequisites: 655 656 1. Ethtool support in the base drivers for retrieving 657 the speed and duplex of each slave. 658 659 2. A switch that supports IEEE 802.3ad Dynamic link 660 aggregation. 661 662 Most switches will require some type of configuration 663 to enable 802.3ad mode. 664 665 balance-tlb or 5 666 667 Adaptive transmit load balancing: channel bonding that 668 does not require any special switch support. 669 670 In tlb_dynamic_lb=1 mode; the outgoing traffic is 671 distributed according to the current load (computed 672 relative to the speed) on each slave. 673 674 In tlb_dynamic_lb=0 mode; the load balancing based on 675 current load is disabled and the load is distributed 676 only using the hash distribution. 677 678 Incoming traffic is received by the current slave. 679 If the receiving slave fails, another slave takes over 680 the MAC address of the failed receiving slave. 681 682 Prerequisite: 683 684 Ethtool support in the base drivers for retrieving the 685 speed of each slave. 686 687 balance-alb or 6 688 689 Adaptive load balancing: includes balance-tlb plus 690 receive load balancing (rlb) for IPV4 traffic, and 691 does not require any special switch support. The 692 receive load balancing is achieved by ARP negotiation. 693 The bonding driver intercepts the ARP Replies sent by 694 the local system on their way out and overwrites the 695 source hardware address with the unique hardware 696 address of one of the slaves in the bond such that 697 different peers use different hardware addresses for 698 the server. 699 700 Receive traffic from connections created by the server 701 is also balanced. When the local system sends an ARP 702 Request the bonding driver copies and saves the peer's 703 IP information from the ARP packet. When the ARP 704 Reply arrives from the peer, its hardware address is 705 retrieved and the bonding driver initiates an ARP 706 reply to this peer assigning it to one of the slaves 707 in the bond. A problematic outcome of using ARP 708 negotiation for balancing is that each time that an 709 ARP request is broadcast it uses the hardware address 710 of the bond. Hence, peers learn the hardware address 711 of the bond and the balancing of receive traffic 712 collapses to the current slave. This is handled by 713 sending updates (ARP Replies) to all the peers with 714 their individually assigned hardware address such that 715 the traffic is redistributed. Receive traffic is also 716 redistributed when a new slave is added to the bond 717 and when an inactive slave is re-activated. The 718 receive load is distributed sequentially (round robin) 719 among the group of highest speed slaves in the bond. 720 721 When a link is reconnected or a new slave joins the 722 bond the receive traffic is redistributed among all 723 active slaves in the bond by initiating ARP Replies 724 with the selected MAC address to each of the 725 clients. The updelay parameter (detailed below) must 726 be set to a value equal or greater than the switch's 727 forwarding delay so that the ARP Replies sent to the 728 peers will not be blocked by the switch. 729 730 Prerequisites: 731 732 1. Ethtool support in the base drivers for retrieving 733 the speed of each slave. 734 735 2. Base driver support for setting the hardware 736 address of a device while it is open. This is 737 required so that there will always be one slave in the 738 team using the bond hardware address (the 739 curr_active_slave) while having a unique hardware 740 address for each slave in the bond. If the 741 curr_active_slave fails its hardware address is 742 swapped with the new curr_active_slave that was 743 chosen. 744 745num_grat_arp, 746num_unsol_na 747 748 Specify the number of peer notifications (gratuitous ARPs and 749 unsolicited IPv6 Neighbor Advertisements) to be issued after a 750 failover event. As soon as the link is up on the new slave 751 (possibly immediately) a peer notification is sent on the 752 bonding device and each VLAN sub-device. This is repeated at 753 the rate specified by peer_notif_delay if the number is 754 greater than 1. 755 756 The valid range is 0 - 255; the default value is 1. These options 757 affect only the active-backup mode. These options were added for 758 bonding versions 3.3.0 and 3.4.0 respectively. 759 760 From Linux 3.0 and bonding version 3.7.1, these notifications 761 are generated by the ipv4 and ipv6 code and the numbers of 762 repetitions cannot be set independently. 763 764packets_per_slave 765 766 Specify the number of packets to transmit through a slave before 767 moving to the next one. When set to 0 then a slave is chosen at 768 random. 769 770 The valid range is 0 - 65535; the default value is 1. This option 771 has effect only in balance-rr mode. 772 773peer_notif_delay 774 775 Specify the delay, in milliseconds, between each peer 776 notification (gratuitous ARP and unsolicited IPv6 Neighbor 777 Advertisement) when they are issued after a failover event. 778 This delay should be a multiple of the link monitor interval 779 (arp_interval or miimon, whichever is active). The default 780 value is 0 which means to match the value of the link monitor 781 interval. 782 783prio 784 Slave priority. A higher number means higher priority. 785 The primary slave has the highest priority. This option also 786 follows the primary_reselect rules. 787 788 This option could only be configured via netlink, and is only valid 789 for active-backup(1), balance-tlb (5) and balance-alb (6) mode. 790 The valid value range is a signed 32 bit integer. 791 792 The default value is 0. 793 794primary 795 796 A string (eth0, eth2, etc) specifying which slave is the 797 primary device. The specified device will always be the 798 active slave while it is available. Only when the primary is 799 off-line will alternate devices be used. This is useful when 800 one slave is preferred over another, e.g., when one slave has 801 higher throughput than another. 802 803 The primary option is only valid for active-backup(1), 804 balance-tlb (5) and balance-alb (6) mode. 805 806primary_reselect 807 808 Specifies the reselection policy for the primary slave. This 809 affects how the primary slave is chosen to become the active slave 810 when failure of the active slave or recovery of the primary slave 811 occurs. This option is designed to prevent flip-flopping between 812 the primary slave and other slaves. Possible values are: 813 814 always or 0 (default) 815 816 The primary slave becomes the active slave whenever it 817 comes back up. 818 819 better or 1 820 821 The primary slave becomes the active slave when it comes 822 back up, if the speed and duplex of the primary slave is 823 better than the speed and duplex of the current active 824 slave. 825 826 failure or 2 827 828 The primary slave becomes the active slave only if the 829 current active slave fails and the primary slave is up. 830 831 The primary_reselect setting is ignored in two cases: 832 833 If no slaves are active, the first slave to recover is 834 made the active slave. 835 836 When initially enslaved, the primary slave is always made 837 the active slave. 838 839 Changing the primary_reselect policy via sysfs will cause an 840 immediate selection of the best active slave according to the new 841 policy. This may or may not result in a change of the active 842 slave, depending upon the circumstances. 843 844 This option was added for bonding version 3.6.0. 845 846tlb_dynamic_lb 847 848 Specifies if dynamic shuffling of flows is enabled in tlb 849 mode. The value has no effect on any other modes. 850 851 The default behavior of tlb mode is to shuffle active flows across 852 slaves based on the load in that interval. This gives nice lb 853 characteristics but can cause packet reordering. If re-ordering is 854 a concern use this variable to disable flow shuffling and rely on 855 load balancing provided solely by the hash distribution. 856 xmit-hash-policy can be used to select the appropriate hashing for 857 the setup. 858 859 The sysfs entry can be used to change the setting per bond device 860 and the initial value is derived from the module parameter. The 861 sysfs entry is allowed to be changed only if the bond device is 862 down. 863 864 The default value is "1" that enables flow shuffling while value "0" 865 disables it. This option was added in bonding driver 3.7.1 866 867 868updelay 869 870 Specifies the time, in milliseconds, to wait before enabling a 871 slave after a link recovery has been detected. This option is 872 only valid for the miimon link monitor. The updelay value 873 should be a multiple of the miimon value; if not, it will be 874 rounded down to the nearest multiple. The default value is 0. 875 876use_carrier 877 878 Specifies whether or not miimon should use MII or ETHTOOL 879 ioctls vs. netif_carrier_ok() to determine the link 880 status. The MII or ETHTOOL ioctls are less efficient and 881 utilize a deprecated calling sequence within the kernel. The 882 netif_carrier_ok() relies on the device driver to maintain its 883 state with netif_carrier_on/off; at this writing, most, but 884 not all, device drivers support this facility. 885 886 If bonding insists that the link is up when it should not be, 887 it may be that your network device driver does not support 888 netif_carrier_on/off. The default state for netif_carrier is 889 "carrier on," so if a driver does not support netif_carrier, 890 it will appear as if the link is always up. In this case, 891 setting use_carrier to 0 will cause bonding to revert to the 892 MII / ETHTOOL ioctl method to determine the link state. 893 894 A value of 1 enables the use of netif_carrier_ok(), a value of 895 0 will use the deprecated MII / ETHTOOL ioctls. The default 896 value is 1. 897 898xmit_hash_policy 899 900 Selects the transmit hash policy to use for slave selection in 901 balance-xor, 802.3ad, and tlb modes. Possible values are: 902 903 layer2 904 905 Uses XOR of hardware MAC addresses and packet type ID 906 field to generate the hash. The formula is 907 908 hash = source MAC[5] XOR destination MAC[5] XOR packet type ID 909 slave number = hash modulo slave count 910 911 This algorithm will place all traffic to a particular 912 network peer on the same slave. 913 914 This algorithm is 802.3ad compliant. 915 916 layer2+3 917 918 This policy uses a combination of layer2 and layer3 919 protocol information to generate the hash. 920 921 Uses XOR of hardware MAC addresses and IP addresses to 922 generate the hash. The formula is 923 924 hash = source MAC[5] XOR destination MAC[5] XOR packet type ID 925 hash = hash XOR source IP XOR destination IP 926 hash = hash XOR (hash RSHIFT 16) 927 hash = hash XOR (hash RSHIFT 8) 928 And then hash is reduced modulo slave count. 929 930 If the protocol is IPv6 then the source and destination 931 addresses are first hashed using ipv6_addr_hash. 932 933 This algorithm will place all traffic to a particular 934 network peer on the same slave. For non-IP traffic, 935 the formula is the same as for the layer2 transmit 936 hash policy. 937 938 This policy is intended to provide a more balanced 939 distribution of traffic than layer2 alone, especially 940 in environments where a layer3 gateway device is 941 required to reach most destinations. 942 943 This algorithm is 802.3ad compliant. 944 945 layer3+4 946 947 This policy uses upper layer protocol information, 948 when available, to generate the hash. This allows for 949 traffic to a particular network peer to span multiple 950 slaves, although a single connection will not span 951 multiple slaves. 952 953 The formula for unfragmented TCP and UDP packets is 954 955 hash = source port, destination port (as in the header) 956 hash = hash XOR source IP XOR destination IP 957 hash = hash XOR (hash RSHIFT 16) 958 hash = hash XOR (hash RSHIFT 8) 959 And then hash is reduced modulo slave count. 960 961 If the protocol is IPv6 then the source and destination 962 addresses are first hashed using ipv6_addr_hash. 963 964 For fragmented TCP or UDP packets and all other IPv4 and 965 IPv6 protocol traffic, the source and destination port 966 information is omitted. For non-IP traffic, the 967 formula is the same as for the layer2 transmit hash 968 policy. 969 970 This algorithm is not fully 802.3ad compliant. A 971 single TCP or UDP conversation containing both 972 fragmented and unfragmented packets will see packets 973 striped across two interfaces. This may result in out 974 of order delivery. Most traffic types will not meet 975 this criteria, as TCP rarely fragments traffic, and 976 most UDP traffic is not involved in extended 977 conversations. Other implementations of 802.3ad may 978 or may not tolerate this noncompliance. 979 980 encap2+3 981 982 This policy uses the same formula as layer2+3 but it 983 relies on skb_flow_dissect to obtain the header fields 984 which might result in the use of inner headers if an 985 encapsulation protocol is used. For example this will 986 improve the performance for tunnel users because the 987 packets will be distributed according to the encapsulated 988 flows. 989 990 encap3+4 991 992 This policy uses the same formula as layer3+4 but it 993 relies on skb_flow_dissect to obtain the header fields 994 which might result in the use of inner headers if an 995 encapsulation protocol is used. For example this will 996 improve the performance for tunnel users because the 997 packets will be distributed according to the encapsulated 998 flows. 999 1000 vlan+srcmac 1001 1002 This policy uses a very rudimentary vlan ID and source mac 1003 hash to load-balance traffic per-vlan, with failover 1004 should one leg fail. The intended use case is for a bond 1005 shared by multiple virtual machines, all configured to 1006 use their own vlan, to give lacp-like functionality 1007 without requiring lacp-capable switching hardware. 1008 1009 The formula for the hash is simply 1010 1011 hash = (vlan ID) XOR (source MAC vendor) XOR (source MAC dev) 1012 1013 The default value is layer2. This option was added in bonding 1014 version 2.6.3. In earlier versions of bonding, this parameter 1015 does not exist, and the layer2 policy is the only policy. The 1016 layer2+3 value was added for bonding version 3.2.2. 1017 1018resend_igmp 1019 1020 Specifies the number of IGMP membership reports to be issued after 1021 a failover event. One membership report is issued immediately after 1022 the failover, subsequent packets are sent in each 200ms interval. 1023 1024 The valid range is 0 - 255; the default value is 1. A value of 0 1025 prevents the IGMP membership report from being issued in response 1026 to the failover event. 1027 1028 This option is useful for bonding modes balance-rr (0), active-backup 1029 (1), balance-tlb (5) and balance-alb (6), in which a failover can 1030 switch the IGMP traffic from one slave to another. Therefore a fresh 1031 IGMP report must be issued to cause the switch to forward the incoming 1032 IGMP traffic over the newly selected slave. 1033 1034 This option was added for bonding version 3.7.0. 1035 1036lp_interval 1037 1038 Specifies the number of seconds between instances where the bonding 1039 driver sends learning packets to each slaves peer switch. 1040 1041 The valid range is 1 - 0x7fffffff; the default value is 1. This Option 1042 has effect only in balance-tlb and balance-alb modes. 1043 10443. Configuring Bonding Devices 1045============================== 1046 1047You can configure bonding using either your distro's network 1048initialization scripts, or manually using either iproute2 or the 1049sysfs interface. Distros generally use one of three packages for the 1050network initialization scripts: initscripts, sysconfig or interfaces. 1051Recent versions of these packages have support for bonding, while older 1052versions do not. 1053 1054We will first describe the options for configuring bonding for 1055distros using versions of initscripts, sysconfig and interfaces with full 1056or partial support for bonding, then provide information on enabling 1057bonding without support from the network initialization scripts (i.e., 1058older versions of initscripts or sysconfig). 1059 1060If you're unsure whether your distro uses sysconfig, 1061initscripts or interfaces, or don't know if it's new enough, have no fear. 1062Determining this is fairly straightforward. 1063 1064First, look for a file called interfaces in /etc/network directory. 1065If this file is present in your system, then your system use interfaces. See 1066Configuration with Interfaces Support. 1067 1068Else, issue the command:: 1069 1070 $ rpm -qf /sbin/ifup 1071 1072It will respond with a line of text starting with either 1073"initscripts" or "sysconfig," followed by some numbers. This is the 1074package that provides your network initialization scripts. 1075 1076Next, to determine if your installation supports bonding, 1077issue the command:: 1078 1079 $ grep ifenslave /sbin/ifup 1080 1081If this returns any matches, then your initscripts or 1082sysconfig has support for bonding. 1083 10843.1 Configuration with Sysconfig Support 1085---------------------------------------- 1086 1087This section applies to distros using a version of sysconfig 1088with bonding support, for example, SuSE Linux Enterprise Server 9. 1089 1090SuSE SLES 9's networking configuration system does support 1091bonding, however, at this writing, the YaST system configuration 1092front end does not provide any means to work with bonding devices. 1093Bonding devices can be managed by hand, however, as follows. 1094 1095First, if they have not already been configured, configure the 1096slave devices. On SLES 9, this is most easily done by running the 1097yast2 sysconfig configuration utility. The goal is for to create an 1098ifcfg-id file for each slave device. The simplest way to accomplish 1099this is to configure the devices for DHCP (this is only to get the 1100file ifcfg-id file created; see below for some issues with DHCP). The 1101name of the configuration file for each device will be of the form:: 1102 1103 ifcfg-id-xx:xx:xx:xx:xx:xx 1104 1105Where the "xx" portion will be replaced with the digits from 1106the device's permanent MAC address. 1107 1108Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been 1109created, it is necessary to edit the configuration files for the slave 1110devices (the MAC addresses correspond to those of the slave devices). 1111Before editing, the file will contain multiple lines, and will look 1112something like this:: 1113 1114 BOOTPROTO='dhcp' 1115 STARTMODE='on' 1116 USERCTL='no' 1117 UNIQUE='XNzu.WeZGOGF+4wE' 1118 _nm_name='bus-pci-0001:61:01.0' 1119 1120Change the BOOTPROTO and STARTMODE lines to the following:: 1121 1122 BOOTPROTO='none' 1123 STARTMODE='off' 1124 1125Do not alter the UNIQUE or _nm_name lines. Remove any other 1126lines (USERCTL, etc). 1127 1128Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified, 1129it's time to create the configuration file for the bonding device 1130itself. This file is named ifcfg-bondX, where X is the number of the 1131bonding device to create, starting at 0. The first such file is 1132ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig 1133network configuration system will correctly start multiple instances 1134of bonding. 1135 1136The contents of the ifcfg-bondX file is as follows:: 1137 1138 BOOTPROTO="static" 1139 BROADCAST="10.0.2.255" 1140 IPADDR="10.0.2.10" 1141 NETMASK="255.255.0.0" 1142 NETWORK="10.0.2.0" 1143 REMOTE_IPADDR="" 1144 STARTMODE="onboot" 1145 BONDING_MASTER="yes" 1146 BONDING_MODULE_OPTS="mode=active-backup miimon=100" 1147 BONDING_SLAVE0="eth0" 1148 BONDING_SLAVE1="bus-pci-0000:06:08.1" 1149 1150Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK 1151values with the appropriate values for your network. 1152 1153The STARTMODE specifies when the device is brought online. 1154The possible values are: 1155 1156 ======== ====================================================== 1157 onboot The device is started at boot time. If you're not 1158 sure, this is probably what you want. 1159 1160 manual The device is started only when ifup is called 1161 manually. Bonding devices may be configured this 1162 way if you do not wish them to start automatically 1163 at boot for some reason. 1164 1165 hotplug The device is started by a hotplug event. This is not 1166 a valid choice for a bonding device. 1167 1168 off or The device configuration is ignored. 1169 ignore 1170 ======== ====================================================== 1171 1172The line BONDING_MASTER='yes' indicates that the device is a 1173bonding master device. The only useful value is "yes." 1174 1175The contents of BONDING_MODULE_OPTS are supplied to the 1176instance of the bonding module for this device. Specify the options 1177for the bonding mode, link monitoring, and so on here. Do not include 1178the max_bonds bonding parameter; this will confuse the configuration 1179system if you have multiple bonding devices. 1180 1181Finally, supply one BONDING_SLAVEn="slave device" for each 1182slave. where "n" is an increasing value, one for each slave. The 1183"slave device" is either an interface name, e.g., "eth0", or a device 1184specifier for the network device. The interface name is easier to 1185find, but the ethN names are subject to change at boot time if, e.g., 1186a device early in the sequence has failed. The device specifiers 1187(bus-pci-0000:06:08.1 in the example above) specify the physical 1188network device, and will not change unless the device's bus location 1189changes (for example, it is moved from one PCI slot to another). The 1190example above uses one of each type for demonstration purposes; most 1191configurations will choose one or the other for all slave devices. 1192 1193When all configuration files have been modified or created, 1194networking must be restarted for the configuration changes to take 1195effect. This can be accomplished via the following:: 1196 1197 # /etc/init.d/network restart 1198 1199Note that the network control script (/sbin/ifdown) will 1200remove the bonding module as part of the network shutdown processing, 1201so it is not necessary to remove the module by hand if, e.g., the 1202module parameters have changed. 1203 1204Also, at this writing, YaST/YaST2 will not manage bonding 1205devices (they do not show bonding interfaces on its list of network 1206devices). It is necessary to edit the configuration file by hand to 1207change the bonding configuration. 1208 1209Additional general options and details of the ifcfg file 1210format can be found in an example ifcfg template file:: 1211 1212 /etc/sysconfig/network/ifcfg.template 1213 1214Note that the template does not document the various ``BONDING_*`` 1215settings described above, but does describe many of the other options. 1216 12173.1.1 Using DHCP with Sysconfig 1218------------------------------- 1219 1220Under sysconfig, configuring a device with BOOTPROTO='dhcp' 1221will cause it to query DHCP for its IP address information. At this 1222writing, this does not function for bonding devices; the scripts 1223attempt to obtain the device address from DHCP prior to adding any of 1224the slave devices. Without active slaves, the DHCP requests are not 1225sent to the network. 1226 12273.1.2 Configuring Multiple Bonds with Sysconfig 1228----------------------------------------------- 1229 1230The sysconfig network initialization system is capable of 1231handling multiple bonding devices. All that is necessary is for each 1232bonding instance to have an appropriately configured ifcfg-bondX file 1233(as described above). Do not specify the "max_bonds" parameter to any 1234instance of bonding, as this will confuse sysconfig. If you require 1235multiple bonding devices with identical parameters, create multiple 1236ifcfg-bondX files. 1237 1238Because the sysconfig scripts supply the bonding module 1239options in the ifcfg-bondX file, it is not necessary to add them to 1240the system ``/etc/modules.d/*.conf`` configuration files. 1241 12423.2 Configuration with Initscripts Support 1243------------------------------------------ 1244 1245This section applies to distros using a recent version of 1246initscripts with bonding support, for example, Red Hat Enterprise Linux 1247version 3 or later, Fedora, etc. On these systems, the network 1248initialization scripts have knowledge of bonding, and can be configured to 1249control bonding devices. Note that older versions of the initscripts 1250package have lower levels of support for bonding; this will be noted where 1251applicable. 1252 1253These distros will not automatically load the network adapter 1254driver unless the ethX device is configured with an IP address. 1255Because of this constraint, users must manually configure a 1256network-script file for all physical adapters that will be members of 1257a bondX link. Network script files are located in the directory: 1258 1259/etc/sysconfig/network-scripts 1260 1261The file name must be prefixed with "ifcfg-eth" and suffixed 1262with the adapter's physical adapter number. For example, the script 1263for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0. 1264Place the following text in the file:: 1265 1266 DEVICE=eth0 1267 USERCTL=no 1268 ONBOOT=yes 1269 MASTER=bond0 1270 SLAVE=yes 1271 BOOTPROTO=none 1272 1273The DEVICE= line will be different for every ethX device and 1274must correspond with the name of the file, i.e., ifcfg-eth1 must have 1275a device line of DEVICE=eth1. The setting of the MASTER= line will 1276also depend on the final bonding interface name chosen for your bond. 1277As with other network devices, these typically start at 0, and go up 1278one for each device, i.e., the first bonding instance is bond0, the 1279second is bond1, and so on. 1280 1281Next, create a bond network script. The file name for this 1282script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is 1283the number of the bond. For bond0 the file is named "ifcfg-bond0", 1284for bond1 it is named "ifcfg-bond1", and so on. Within that file, 1285place the following text:: 1286 1287 DEVICE=bond0 1288 IPADDR=192.168.1.1 1289 NETMASK=255.255.255.0 1290 NETWORK=192.168.1.0 1291 BROADCAST=192.168.1.255 1292 ONBOOT=yes 1293 BOOTPROTO=none 1294 USERCTL=no 1295 1296Be sure to change the networking specific lines (IPADDR, 1297NETMASK, NETWORK and BROADCAST) to match your network configuration. 1298 1299For later versions of initscripts, such as that found with Fedora 13007 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible, 1301and, indeed, preferable, to specify the bonding options in the ifcfg-bond0 1302file, e.g. a line of the format:: 1303 1304 BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254" 1305 1306will configure the bond with the specified options. The options 1307specified in BONDING_OPTS are identical to the bonding module parameters 1308except for the arp_ip_target field when using versions of initscripts older 1309than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2). When 1310using older versions each target should be included as a separate option and 1311should be preceded by a '+' to indicate it should be added to the list of 1312queried targets, e.g.,:: 1313 1314 arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2 1315 1316is the proper syntax to specify multiple targets. When specifying 1317options via BONDING_OPTS, it is not necessary to edit 1318``/etc/modprobe.d/*.conf``. 1319 1320For even older versions of initscripts that do not support 1321BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon 1322your distro) to load the bonding module with your desired options when the 1323bond0 interface is brought up. The following lines in /etc/modprobe.d/*.conf 1324will load the bonding module, and select its options: 1325 1326 alias bond0 bonding 1327 options bond0 mode=balance-alb miimon=100 1328 1329Replace the sample parameters with the appropriate set of 1330options for your configuration. 1331 1332Finally run "/etc/rc.d/init.d/network restart" as root. This 1333will restart the networking subsystem and your bond link should be now 1334up and running. 1335 13363.2.1 Using DHCP with Initscripts 1337--------------------------------- 1338 1339Recent versions of initscripts (the versions supplied with Fedora 1340Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to 1341work) have support for assigning IP information to bonding devices via 1342DHCP. 1343 1344To configure bonding for DHCP, configure it as described 1345above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp" 1346and add a line consisting of "TYPE=Bonding". Note that the TYPE value 1347is case sensitive. 1348 13493.2.2 Configuring Multiple Bonds with Initscripts 1350------------------------------------------------- 1351 1352Initscripts packages that are included with Fedora 7 and Red Hat 1353Enterprise Linux 5 support multiple bonding interfaces by simply 1354specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the 1355number of the bond. This support requires sysfs support in the kernel, 1356and a bonding driver of version 3.0.0 or later. Other configurations may 1357not support this method for specifying multiple bonding interfaces; for 1358those instances, see the "Configuring Multiple Bonds Manually" section, 1359below. 1360 13613.3 Configuring Bonding Manually with iproute2 1362----------------------------------------------- 1363 1364This section applies to distros whose network initialization 1365scripts (the sysconfig or initscripts package) do not have specific 1366knowledge of bonding. One such distro is SuSE Linux Enterprise Server 1367version 8. 1368 1369The general method for these systems is to place the bonding 1370module parameters into a config file in /etc/modprobe.d/ (as 1371appropriate for the installed distro), then add modprobe and/or 1372`ip link` commands to the system's global init script. The name of 1373the global init script differs; for sysconfig, it is 1374/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local. 1375 1376For example, if you wanted to make a simple bond of two e100 1377devices (presumed to be eth0 and eth1), and have it persist across 1378reboots, edit the appropriate file (/etc/init.d/boot.local or 1379/etc/rc.d/rc.local), and add the following:: 1380 1381 modprobe bonding mode=balance-alb miimon=100 1382 modprobe e100 1383 ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up 1384 ip link set eth0 master bond0 1385 ip link set eth1 master bond0 1386 1387Replace the example bonding module parameters and bond0 1388network configuration (IP address, netmask, etc) with the appropriate 1389values for your configuration. 1390 1391Unfortunately, this method will not provide support for the 1392ifup and ifdown scripts on the bond devices. To reload the bonding 1393configuration, it is necessary to run the initialization script, e.g.,:: 1394 1395 # /etc/init.d/boot.local 1396 1397or:: 1398 1399 # /etc/rc.d/rc.local 1400 1401It may be desirable in such a case to create a separate script 1402which only initializes the bonding configuration, then call that 1403separate script from within boot.local. This allows for bonding to be 1404enabled without re-running the entire global init script. 1405 1406To shut down the bonding devices, it is necessary to first 1407mark the bonding device itself as being down, then remove the 1408appropriate device driver modules. For our example above, you can do 1409the following:: 1410 1411 # ifconfig bond0 down 1412 # rmmod bonding 1413 # rmmod e100 1414 1415Again, for convenience, it may be desirable to create a script 1416with these commands. 1417 1418 14193.3.1 Configuring Multiple Bonds Manually 1420----------------------------------------- 1421 1422This section contains information on configuring multiple 1423bonding devices with differing options for those systems whose network 1424initialization scripts lack support for configuring multiple bonds. 1425 1426If you require multiple bonding devices, but all with the same 1427options, you may wish to use the "max_bonds" module parameter, 1428documented above. 1429 1430To create multiple bonding devices with differing options, it is 1431preferable to use bonding parameters exported by sysfs, documented in the 1432section below. 1433 1434For versions of bonding without sysfs support, the only means to 1435provide multiple instances of bonding with differing options is to load 1436the bonding driver multiple times. Note that current versions of the 1437sysconfig network initialization scripts handle this automatically; if 1438your distro uses these scripts, no special action is needed. See the 1439section Configuring Bonding Devices, above, if you're not sure about your 1440network initialization scripts. 1441 1442To load multiple instances of the module, it is necessary to 1443specify a different name for each instance (the module loading system 1444requires that every loaded module, even multiple instances of the same 1445module, have a unique name). This is accomplished by supplying multiple 1446sets of bonding options in ``/etc/modprobe.d/*.conf``, for example:: 1447 1448 alias bond0 bonding 1449 options bond0 -o bond0 mode=balance-rr miimon=100 1450 1451 alias bond1 bonding 1452 options bond1 -o bond1 mode=balance-alb miimon=50 1453 1454will load the bonding module two times. The first instance is 1455named "bond0" and creates the bond0 device in balance-rr mode with an 1456miimon of 100. The second instance is named "bond1" and creates the 1457bond1 device in balance-alb mode with an miimon of 50. 1458 1459In some circumstances (typically with older distributions), 1460the above does not work, and the second bonding instance never sees 1461its options. In that case, the second options line can be substituted 1462as follows:: 1463 1464 install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \ 1465 mode=balance-alb miimon=50 1466 1467This may be repeated any number of times, specifying a new and 1468unique name in place of bond1 for each subsequent instance. 1469 1470It has been observed that some Red Hat supplied kernels are unable 1471to rename modules at load time (the "-o bond1" part). Attempts to pass 1472that option to modprobe will produce an "Operation not permitted" error. 1473This has been reported on some Fedora Core kernels, and has been seen on 1474RHEL 4 as well. On kernels exhibiting this problem, it will be impossible 1475to configure multiple bonds with differing parameters (as they are older 1476kernels, and also lack sysfs support). 1477 14783.4 Configuring Bonding Manually via Sysfs 1479------------------------------------------ 1480 1481Starting with version 3.0.0, Channel Bonding may be configured 1482via the sysfs interface. This interface allows dynamic configuration 1483of all bonds in the system without unloading the module. It also 1484allows for adding and removing bonds at runtime. Ifenslave is no 1485longer required, though it is still supported. 1486 1487Use of the sysfs interface allows you to use multiple bonds 1488with different configurations without having to reload the module. 1489It also allows you to use multiple, differently configured bonds when 1490bonding is compiled into the kernel. 1491 1492You must have the sysfs filesystem mounted to configure 1493bonding this way. The examples in this document assume that you 1494are using the standard mount point for sysfs, e.g. /sys. If your 1495sysfs filesystem is mounted elsewhere, you will need to adjust the 1496example paths accordingly. 1497 1498Creating and Destroying Bonds 1499----------------------------- 1500To add a new bond foo:: 1501 1502 # echo +foo > /sys/class/net/bonding_masters 1503 1504To remove an existing bond bar:: 1505 1506 # echo -bar > /sys/class/net/bonding_masters 1507 1508To show all existing bonds:: 1509 1510 # cat /sys/class/net/bonding_masters 1511 1512.. note:: 1513 1514 due to 4K size limitation of sysfs files, this list may be 1515 truncated if you have more than a few hundred bonds. This is unlikely 1516 to occur under normal operating conditions. 1517 1518Adding and Removing Slaves 1519-------------------------- 1520Interfaces may be enslaved to a bond using the file 1521/sys/class/net/<bond>/bonding/slaves. The semantics for this file 1522are the same as for the bonding_masters file. 1523 1524To enslave interface eth0 to bond bond0:: 1525 1526 # ifconfig bond0 up 1527 # echo +eth0 > /sys/class/net/bond0/bonding/slaves 1528 1529To free slave eth0 from bond bond0:: 1530 1531 # echo -eth0 > /sys/class/net/bond0/bonding/slaves 1532 1533When an interface is enslaved to a bond, symlinks between the 1534two are created in the sysfs filesystem. In this case, you would get 1535/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and 1536/sys/class/net/eth0/master pointing to /sys/class/net/bond0. 1537 1538This means that you can tell quickly whether or not an 1539interface is enslaved by looking for the master symlink. Thus: 1540# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves 1541will free eth0 from whatever bond it is enslaved to, regardless of 1542the name of the bond interface. 1543 1544Changing a Bond's Configuration 1545------------------------------- 1546Each bond may be configured individually by manipulating the 1547files located in /sys/class/net/<bond name>/bonding 1548 1549The names of these files correspond directly with the command- 1550line parameters described elsewhere in this file, and, with the 1551exception of arp_ip_target, they accept the same values. To see the 1552current setting, simply cat the appropriate file. 1553 1554A few examples will be given here; for specific usage 1555guidelines for each parameter, see the appropriate section in this 1556document. 1557 1558To configure bond0 for balance-alb mode:: 1559 1560 # ifconfig bond0 down 1561 # echo 6 > /sys/class/net/bond0/bonding/mode 1562 - or - 1563 # echo balance-alb > /sys/class/net/bond0/bonding/mode 1564 1565.. note:: 1566 1567 The bond interface must be down before the mode can be changed. 1568 1569To enable MII monitoring on bond0 with a 1 second interval:: 1570 1571 # echo 1000 > /sys/class/net/bond0/bonding/miimon 1572 1573.. note:: 1574 1575 If ARP monitoring is enabled, it will disabled when MII 1576 monitoring is enabled, and vice-versa. 1577 1578To add ARP targets:: 1579 1580 # echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target 1581 # echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target 1582 1583.. note:: 1584 1585 up to 16 target addresses may be specified. 1586 1587To remove an ARP target:: 1588 1589 # echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target 1590 1591To configure the interval between learning packet transmits:: 1592 1593 # echo 12 > /sys/class/net/bond0/bonding/lp_interval 1594 1595.. note:: 1596 1597 the lp_interval is the number of seconds between instances where 1598 the bonding driver sends learning packets to each slaves peer switch. The 1599 default interval is 1 second. 1600 1601Example Configuration 1602--------------------- 1603We begin with the same example that is shown in section 3.3, 1604executed with sysfs, and without using ifenslave. 1605 1606To make a simple bond of two e100 devices (presumed to be eth0 1607and eth1), and have it persist across reboots, edit the appropriate 1608file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the 1609following:: 1610 1611 modprobe bonding 1612 modprobe e100 1613 echo balance-alb > /sys/class/net/bond0/bonding/mode 1614 ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up 1615 echo 100 > /sys/class/net/bond0/bonding/miimon 1616 echo +eth0 > /sys/class/net/bond0/bonding/slaves 1617 echo +eth1 > /sys/class/net/bond0/bonding/slaves 1618 1619To add a second bond, with two e1000 interfaces in 1620active-backup mode, using ARP monitoring, add the following lines to 1621your init script:: 1622 1623 modprobe e1000 1624 echo +bond1 > /sys/class/net/bonding_masters 1625 echo active-backup > /sys/class/net/bond1/bonding/mode 1626 ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up 1627 echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target 1628 echo 2000 > /sys/class/net/bond1/bonding/arp_interval 1629 echo +eth2 > /sys/class/net/bond1/bonding/slaves 1630 echo +eth3 > /sys/class/net/bond1/bonding/slaves 1631 16323.5 Configuration with Interfaces Support 1633----------------------------------------- 1634 1635This section applies to distros which use /etc/network/interfaces file 1636to describe network interface configuration, most notably Debian and it's 1637derivatives. 1638 1639The ifup and ifdown commands on Debian don't support bonding out of 1640the box. The ifenslave-2.6 package should be installed to provide bonding 1641support. Once installed, this package will provide ``bond-*`` options 1642to be used into /etc/network/interfaces. 1643 1644Note that ifenslave-2.6 package will load the bonding module and use 1645the ifenslave command when appropriate. 1646 1647Example Configurations 1648---------------------- 1649 1650In /etc/network/interfaces, the following stanza will configure bond0, in 1651active-backup mode, with eth0 and eth1 as slaves:: 1652 1653 auto bond0 1654 iface bond0 inet dhcp 1655 bond-slaves eth0 eth1 1656 bond-mode active-backup 1657 bond-miimon 100 1658 bond-primary eth0 eth1 1659 1660If the above configuration doesn't work, you might have a system using 1661upstart for system startup. This is most notably true for recent 1662Ubuntu versions. The following stanza in /etc/network/interfaces will 1663produce the same result on those systems:: 1664 1665 auto bond0 1666 iface bond0 inet dhcp 1667 bond-slaves none 1668 bond-mode active-backup 1669 bond-miimon 100 1670 1671 auto eth0 1672 iface eth0 inet manual 1673 bond-master bond0 1674 bond-primary eth0 eth1 1675 1676 auto eth1 1677 iface eth1 inet manual 1678 bond-master bond0 1679 bond-primary eth0 eth1 1680 1681For a full list of ``bond-*`` supported options in /etc/network/interfaces and 1682some more advanced examples tailored to you particular distros, see the files in 1683/usr/share/doc/ifenslave-2.6. 1684 16853.6 Overriding Configuration for Special Cases 1686---------------------------------------------- 1687 1688When using the bonding driver, the physical port which transmits a frame is 1689typically selected by the bonding driver, and is not relevant to the user or 1690system administrator. The output port is simply selected using the policies of 1691the selected bonding mode. On occasion however, it is helpful to direct certain 1692classes of traffic to certain physical interfaces on output to implement 1693slightly more complex policies. For example, to reach a web server over a 1694bonded interface in which eth0 connects to a private network, while eth1 1695connects via a public network, it may be desirous to bias the bond to send said 1696traffic over eth0 first, using eth1 only as a fall back, while all other traffic 1697can safely be sent over either interface. Such configurations may be achieved 1698using the traffic control utilities inherent in linux. 1699 1700By default the bonding driver is multiqueue aware and 16 queues are created 1701when the driver initializes (see Documentation/networking/multiqueue.rst 1702for details). If more or less queues are desired the module parameter 1703tx_queues can be used to change this value. There is no sysfs parameter 1704available as the allocation is done at module init time. 1705 1706The output of the file /proc/net/bonding/bondX has changed so the output Queue 1707ID is now printed for each slave:: 1708 1709 Bonding Mode: fault-tolerance (active-backup) 1710 Primary Slave: None 1711 Currently Active Slave: eth0 1712 MII Status: up 1713 MII Polling Interval (ms): 0 1714 Up Delay (ms): 0 1715 Down Delay (ms): 0 1716 1717 Slave Interface: eth0 1718 MII Status: up 1719 Link Failure Count: 0 1720 Permanent HW addr: 00:1a:a0:12:8f:cb 1721 Slave queue ID: 0 1722 1723 Slave Interface: eth1 1724 MII Status: up 1725 Link Failure Count: 0 1726 Permanent HW addr: 00:1a:a0:12:8f:cc 1727 Slave queue ID: 2 1728 1729The queue_id for a slave can be set using the command:: 1730 1731 # echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id 1732 1733Any interface that needs a queue_id set should set it with multiple calls 1734like the one above until proper priorities are set for all interfaces. On 1735distributions that allow configuration via initscripts, multiple 'queue_id' 1736arguments can be added to BONDING_OPTS to set all needed slave queues. 1737 1738These queue id's can be used in conjunction with the tc utility to configure 1739a multiqueue qdisc and filters to bias certain traffic to transmit on certain 1740slave devices. For instance, say we wanted, in the above configuration to 1741force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output 1742device. The following commands would accomplish this:: 1743 1744 # tc qdisc add dev bond0 handle 1 root multiq 1745 1746 # tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip \ 1747 dst 192.168.1.100 action skbedit queue_mapping 2 1748 1749These commands tell the kernel to attach a multiqueue queue discipline to the 1750bond0 interface and filter traffic enqueued to it, such that packets with a dst 1751ip of 192.168.1.100 have their output queue mapping value overwritten to 2. 1752This value is then passed into the driver, causing the normal output path 1753selection policy to be overridden, selecting instead qid 2, which maps to eth1. 1754 1755Note that qid values begin at 1. Qid 0 is reserved to initiate to the driver 1756that normal output policy selection should take place. One benefit to simply 1757leaving the qid for a slave to 0 is the multiqueue awareness in the bonding 1758driver that is now present. This awareness allows tc filters to be placed on 1759slave devices as well as bond devices and the bonding driver will simply act as 1760a pass-through for selecting output queues on the slave device rather than 1761output port selection. 1762 1763This feature first appeared in bonding driver version 3.7.0 and support for 1764output slave selection was limited to round-robin and active-backup modes. 1765 17663.7 Configuring LACP for 802.3ad mode in a more secure way 1767---------------------------------------------------------- 1768 1769When using 802.3ad bonding mode, the Actor (host) and Partner (switch) 1770exchange LACPDUs. These LACPDUs cannot be sniffed, because they are 1771destined to link local mac addresses (which switches/bridges are not 1772supposed to forward). However, most of the values are easily predictable 1773or are simply the machine's MAC address (which is trivially known to all 1774other hosts in the same L2). This implies that other machines in the L2 1775domain can spoof LACPDU packets from other hosts to the switch and potentially 1776cause mayhem by joining (from the point of view of the switch) another 1777machine's aggregate, thus receiving a portion of that hosts incoming 1778traffic and / or spoofing traffic from that machine themselves (potentially 1779even successfully terminating some portion of flows). Though this is not 1780a likely scenario, one could avoid this possibility by simply configuring 1781few bonding parameters: 1782 1783 (a) ad_actor_system : You can set a random mac-address that can be used for 1784 these LACPDU exchanges. The value can not be either NULL or Multicast. 1785 Also it's preferable to set the local-admin bit. Following shell code 1786 generates a random mac-address as described above:: 1787 1788 # sys_mac_addr=$(printf '%02x:%02x:%02x:%02x:%02x:%02x' \ 1789 $(( (RANDOM & 0xFE) | 0x02 )) \ 1790 $(( RANDOM & 0xFF )) \ 1791 $(( RANDOM & 0xFF )) \ 1792 $(( RANDOM & 0xFF )) \ 1793 $(( RANDOM & 0xFF )) \ 1794 $(( RANDOM & 0xFF ))) 1795 # echo $sys_mac_addr > /sys/class/net/bond0/bonding/ad_actor_system 1796 1797 (b) ad_actor_sys_prio : Randomize the system priority. The default value 1798 is 65535, but system can take the value from 1 - 65535. Following shell 1799 code generates random priority and sets it:: 1800 1801 # sys_prio=$(( 1 + RANDOM + RANDOM )) 1802 # echo $sys_prio > /sys/class/net/bond0/bonding/ad_actor_sys_prio 1803 1804 (c) ad_user_port_key : Use the user portion of the port-key. The default 1805 keeps this empty. These are the upper 10 bits of the port-key and value 1806 ranges from 0 - 1023. Following shell code generates these 10 bits and 1807 sets it:: 1808 1809 # usr_port_key=$(( RANDOM & 0x3FF )) 1810 # echo $usr_port_key > /sys/class/net/bond0/bonding/ad_user_port_key 1811 1812 18134 Querying Bonding Configuration 1814================================= 1815 18164.1 Bonding Configuration 1817------------------------- 1818 1819Each bonding device has a read-only file residing in the 1820/proc/net/bonding directory. The file contents include information 1821about the bonding configuration, options and state of each slave. 1822 1823For example, the contents of /proc/net/bonding/bond0 after the 1824driver is loaded with parameters of mode=0 and miimon=1000 is 1825generally as follows:: 1826 1827 Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004) 1828 Bonding Mode: load balancing (round-robin) 1829 Currently Active Slave: eth0 1830 MII Status: up 1831 MII Polling Interval (ms): 1000 1832 Up Delay (ms): 0 1833 Down Delay (ms): 0 1834 1835 Slave Interface: eth1 1836 MII Status: up 1837 Link Failure Count: 1 1838 1839 Slave Interface: eth0 1840 MII Status: up 1841 Link Failure Count: 1 1842 1843The precise format and contents will change depending upon the 1844bonding configuration, state, and version of the bonding driver. 1845 18464.2 Network configuration 1847------------------------- 1848 1849The network configuration can be inspected using the ifconfig 1850command. Bonding devices will have the MASTER flag set; Bonding slave 1851devices will have the SLAVE flag set. The ifconfig output does not 1852contain information on which slaves are associated with which masters. 1853 1854In the example below, the bond0 interface is the master 1855(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of 1856bond0 have the same MAC address (HWaddr) as bond0 for all modes except 1857TLB and ALB that require a unique MAC address for each slave:: 1858 1859 # /sbin/ifconfig 1860 bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1861 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0 1862 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 1863 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 1864 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0 1865 collisions:0 txqueuelen:0 1866 1867 eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1868 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 1869 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 1870 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0 1871 collisions:0 txqueuelen:100 1872 Interrupt:10 Base address:0x1080 1873 1874 eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1875 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 1876 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 1877 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 1878 collisions:0 txqueuelen:100 1879 Interrupt:9 Base address:0x1400 1880 18815. Switch Configuration 1882======================= 1883 1884For this section, "switch" refers to whatever system the 1885bonded devices are directly connected to (i.e., where the other end of 1886the cable plugs into). This may be an actual dedicated switch device, 1887or it may be another regular system (e.g., another computer running 1888Linux), 1889 1890The active-backup, balance-tlb and balance-alb modes do not 1891require any specific configuration of the switch. 1892 1893The 802.3ad mode requires that the switch have the appropriate 1894ports configured as an 802.3ad aggregation. The precise method used 1895to configure this varies from switch to switch, but, for example, a 1896Cisco 3550 series switch requires that the appropriate ports first be 1897grouped together in a single etherchannel instance, then that 1898etherchannel is set to mode "lacp" to enable 802.3ad (instead of 1899standard EtherChannel). 1900 1901The balance-rr, balance-xor and broadcast modes generally 1902require that the switch have the appropriate ports grouped together. 1903The nomenclature for such a group differs between switches, it may be 1904called an "etherchannel" (as in the Cisco example, above), a "trunk 1905group" or some other similar variation. For these modes, each switch 1906will also have its own configuration options for the switch's transmit 1907policy to the bond. Typical choices include XOR of either the MAC or 1908IP addresses. The transmit policy of the two peers does not need to 1909match. For these three modes, the bonding mode really selects a 1910transmit policy for an EtherChannel group; all three will interoperate 1911with another EtherChannel group. 1912 1913 19146. 802.1q VLAN Support 1915====================== 1916 1917It is possible to configure VLAN devices over a bond interface 1918using the 8021q driver. However, only packets coming from the 8021q 1919driver and passing through bonding will be tagged by default. Self 1920generated packets, for example, bonding's learning packets or ARP 1921packets generated by either ALB mode or the ARP monitor mechanism, are 1922tagged internally by bonding itself. As a result, bonding must 1923"learn" the VLAN IDs configured above it, and use those IDs to tag 1924self generated packets. 1925 1926For reasons of simplicity, and to support the use of adapters 1927that can do VLAN hardware acceleration offloading, the bonding 1928interface declares itself as fully hardware offloading capable, it gets 1929the add_vid/kill_vid notifications to gather the necessary 1930information, and it propagates those actions to the slaves. In case 1931of mixed adapter types, hardware accelerated tagged packets that 1932should go through an adapter that is not offloading capable are 1933"un-accelerated" by the bonding driver so the VLAN tag sits in the 1934regular location. 1935 1936VLAN interfaces *must* be added on top of a bonding interface 1937only after enslaving at least one slave. The bonding interface has a 1938hardware address of 00:00:00:00:00:00 until the first slave is added. 1939If the VLAN interface is created prior to the first enslavement, it 1940would pick up the all-zeroes hardware address. Once the first slave 1941is attached to the bond, the bond device itself will pick up the 1942slave's hardware address, which is then available for the VLAN device. 1943 1944Also, be aware that a similar problem can occur if all slaves 1945are released from a bond that still has one or more VLAN interfaces on 1946top of it. When a new slave is added, the bonding interface will 1947obtain its hardware address from the first slave, which might not 1948match the hardware address of the VLAN interfaces (which was 1949ultimately copied from an earlier slave). 1950 1951There are two methods to insure that the VLAN device operates 1952with the correct hardware address if all slaves are removed from a 1953bond interface: 1954 19551. Remove all VLAN interfaces then recreate them 1956 19572. Set the bonding interface's hardware address so that it 1958matches the hardware address of the VLAN interfaces. 1959 1960Note that changing a VLAN interface's HW address would set the 1961underlying device -- i.e. the bonding interface -- to promiscuous 1962mode, which might not be what you want. 1963 1964 19657. Link Monitoring 1966================== 1967 1968The bonding driver at present supports two schemes for 1969monitoring a slave device's link state: the ARP monitor and the MII 1970monitor. 1971 1972At the present time, due to implementation restrictions in the 1973bonding driver itself, it is not possible to enable both ARP and MII 1974monitoring simultaneously. 1975 19767.1 ARP Monitor Operation 1977------------------------- 1978 1979The ARP monitor operates as its name suggests: it sends ARP 1980queries to one or more designated peer systems on the network, and 1981uses the response as an indication that the link is operating. This 1982gives some assurance that traffic is actually flowing to and from one 1983or more peers on the local network. 1984 19857.2 Configuring Multiple ARP Targets 1986------------------------------------ 1987 1988While ARP monitoring can be done with just one target, it can 1989be useful in a High Availability setup to have several targets to 1990monitor. In the case of just one target, the target itself may go 1991down or have a problem making it unresponsive to ARP requests. Having 1992an additional target (or several) increases the reliability of the ARP 1993monitoring. 1994 1995Multiple ARP targets must be separated by commas as follows:: 1996 1997 # example options for ARP monitoring with three targets 1998 alias bond0 bonding 1999 options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9 2000 2001For just a single target the options would resemble:: 2002 2003 # example options for ARP monitoring with one target 2004 alias bond0 bonding 2005 options bond0 arp_interval=60 arp_ip_target=192.168.0.100 2006 2007 20087.3 MII Monitor Operation 2009------------------------- 2010 2011The MII monitor monitors only the carrier state of the local 2012network interface. It accomplishes this in one of three ways: by 2013depending upon the device driver to maintain its carrier state, by 2014querying the device's MII registers, or by making an ethtool query to 2015the device. 2016 2017If the use_carrier module parameter is 1 (the default value), 2018then the MII monitor will rely on the driver for carrier state 2019information (via the netif_carrier subsystem). As explained in the 2020use_carrier parameter information, above, if the MII monitor fails to 2021detect carrier loss on the device (e.g., when the cable is physically 2022disconnected), it may be that the driver does not support 2023netif_carrier. 2024 2025If use_carrier is 0, then the MII monitor will first query the 2026device's (via ioctl) MII registers and check the link state. If that 2027request fails (not just that it returns carrier down), then the MII 2028monitor will make an ethtool ETHTOOL_GLINK request to attempt to obtain 2029the same information. If both methods fail (i.e., the driver either 2030does not support or had some error in processing both the MII register 2031and ethtool requests), then the MII monitor will assume the link is 2032up. 2033 20348. Potential Sources of Trouble 2035=============================== 2036 20378.1 Adventures in Routing 2038------------------------- 2039 2040When bonding is configured, it is important that the slave 2041devices not have routes that supersede routes of the master (or, 2042generally, not have routes at all). For example, suppose the bonding 2043device bond0 has two slaves, eth0 and eth1, and the routing table is 2044as follows:: 2045 2046 Kernel IP routing table 2047 Destination Gateway Genmask Flags MSS Window irtt Iface 2048 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0 2049 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1 2050 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0 2051 127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo 2052 2053This routing configuration will likely still update the 2054receive/transmit times in the driver (needed by the ARP monitor), but 2055may bypass the bonding driver (because outgoing traffic to, in this 2056case, another host on network 10 would use eth0 or eth1 before bond0). 2057 2058The ARP monitor (and ARP itself) may become confused by this 2059configuration, because ARP requests (generated by the ARP monitor) 2060will be sent on one interface (bond0), but the corresponding reply 2061will arrive on a different interface (eth0). This reply looks to ARP 2062as an unsolicited ARP reply (because ARP matches replies on an 2063interface basis), and is discarded. The MII monitor is not affected 2064by the state of the routing table. 2065 2066The solution here is simply to insure that slaves do not have 2067routes of their own, and if for some reason they must, those routes do 2068not supersede routes of their master. This should generally be the 2069case, but unusual configurations or errant manual or automatic static 2070route additions may cause trouble. 2071 20728.2 Ethernet Device Renaming 2073---------------------------- 2074 2075On systems with network configuration scripts that do not 2076associate physical devices directly with network interface names (so 2077that the same physical device always has the same "ethX" name), it may 2078be necessary to add some special logic to config files in 2079/etc/modprobe.d/. 2080 2081For example, given a modules.conf containing the following:: 2082 2083 alias bond0 bonding 2084 options bond0 mode=some-mode miimon=50 2085 alias eth0 tg3 2086 alias eth1 tg3 2087 alias eth2 e1000 2088 alias eth3 e1000 2089 2090If neither eth0 and eth1 are slaves to bond0, then when the 2091bond0 interface comes up, the devices may end up reordered. This 2092happens because bonding is loaded first, then its slave device's 2093drivers are loaded next. Since no other drivers have been loaded, 2094when the e1000 driver loads, it will receive eth0 and eth1 for its 2095devices, but the bonding configuration tries to enslave eth2 and eth3 2096(which may later be assigned to the tg3 devices). 2097 2098Adding the following:: 2099 2100 add above bonding e1000 tg3 2101 2102causes modprobe to load e1000 then tg3, in that order, when 2103bonding is loaded. This command is fully documented in the 2104modules.conf manual page. 2105 2106On systems utilizing modprobe an equivalent problem can occur. 2107In this case, the following can be added to config files in 2108/etc/modprobe.d/ as:: 2109 2110 softdep bonding pre: tg3 e1000 2111 2112This will load tg3 and e1000 modules before loading the bonding one. 2113Full documentation on this can be found in the modprobe.d and modprobe 2114manual pages. 2115 21168.3. Painfully Slow Or No Failed Link Detection By Miimon 2117--------------------------------------------------------- 2118 2119By default, bonding enables the use_carrier option, which 2120instructs bonding to trust the driver to maintain carrier state. 2121 2122As discussed in the options section, above, some drivers do 2123not support the netif_carrier_on/_off link state tracking system. 2124With use_carrier enabled, bonding will always see these links as up, 2125regardless of their actual state. 2126 2127Additionally, other drivers do support netif_carrier, but do 2128not maintain it in real time, e.g., only polling the link state at 2129some fixed interval. In this case, miimon will detect failures, but 2130only after some long period of time has expired. If it appears that 2131miimon is very slow in detecting link failures, try specifying 2132use_carrier=0 to see if that improves the failure detection time. If 2133it does, then it may be that the driver checks the carrier state at a 2134fixed interval, but does not cache the MII register values (so the 2135use_carrier=0 method of querying the registers directly works). If 2136use_carrier=0 does not improve the failover, then the driver may cache 2137the registers, or the problem may be elsewhere. 2138 2139Also, remember that miimon only checks for the device's 2140carrier state. It has no way to determine the state of devices on or 2141beyond other ports of a switch, or if a switch is refusing to pass 2142traffic while still maintaining carrier on. 2143 21449. SNMP agents 2145=============== 2146 2147If running SNMP agents, the bonding driver should be loaded 2148before any network drivers participating in a bond. This requirement 2149is due to the interface index (ipAdEntIfIndex) being associated to 2150the first interface found with a given IP address. That is, there is 2151only one ipAdEntIfIndex for each IP address. For example, if eth0 and 2152eth1 are slaves of bond0 and the driver for eth0 is loaded before the 2153bonding driver, the interface for the IP address will be associated 2154with the eth0 interface. This configuration is shown below, the IP 2155address 192.168.1.1 has an interface index of 2 which indexes to eth0 2156in the ifDescr table (ifDescr.2). 2157 2158:: 2159 2160 interfaces.ifTable.ifEntry.ifDescr.1 = lo 2161 interfaces.ifTable.ifEntry.ifDescr.2 = eth0 2162 interfaces.ifTable.ifEntry.ifDescr.3 = eth1 2163 interfaces.ifTable.ifEntry.ifDescr.4 = eth2 2164 interfaces.ifTable.ifEntry.ifDescr.5 = eth3 2165 interfaces.ifTable.ifEntry.ifDescr.6 = bond0 2166 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5 2167 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2 2168 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4 2169 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1 2170 2171This problem is avoided by loading the bonding driver before 2172any network drivers participating in a bond. Below is an example of 2173loading the bonding driver first, the IP address 192.168.1.1 is 2174correctly associated with ifDescr.2. 2175 2176 interfaces.ifTable.ifEntry.ifDescr.1 = lo 2177 interfaces.ifTable.ifEntry.ifDescr.2 = bond0 2178 interfaces.ifTable.ifEntry.ifDescr.3 = eth0 2179 interfaces.ifTable.ifEntry.ifDescr.4 = eth1 2180 interfaces.ifTable.ifEntry.ifDescr.5 = eth2 2181 interfaces.ifTable.ifEntry.ifDescr.6 = eth3 2182 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6 2183 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2 2184 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5 2185 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1 2186 2187While some distributions may not report the interface name in 2188ifDescr, the association between the IP address and IfIndex remains 2189and SNMP functions such as Interface_Scan_Next will report that 2190association. 2191 219210. Promiscuous mode 2193==================== 2194 2195When running network monitoring tools, e.g., tcpdump, it is 2196common to enable promiscuous mode on the device, so that all traffic 2197is seen (instead of seeing only traffic destined for the local host). 2198The bonding driver handles promiscuous mode changes to the bonding 2199master device (e.g., bond0), and propagates the setting to the slave 2200devices. 2201 2202For the balance-rr, balance-xor, broadcast, and 802.3ad modes, 2203the promiscuous mode setting is propagated to all slaves. 2204 2205For the active-backup, balance-tlb and balance-alb modes, the 2206promiscuous mode setting is propagated only to the active slave. 2207 2208For balance-tlb mode, the active slave is the slave currently 2209receiving inbound traffic. 2210 2211For balance-alb mode, the active slave is the slave used as a 2212"primary." This slave is used for mode-specific control traffic, for 2213sending to peers that are unassigned or if the load is unbalanced. 2214 2215For the active-backup, balance-tlb and balance-alb modes, when 2216the active slave changes (e.g., due to a link failure), the 2217promiscuous setting will be propagated to the new active slave. 2218 221911. Configuring Bonding for High Availability 2220============================================= 2221 2222High Availability refers to configurations that provide 2223maximum network availability by having redundant or backup devices, 2224links or switches between the host and the rest of the world. The 2225goal is to provide the maximum availability of network connectivity 2226(i.e., the network always works), even though other configurations 2227could provide higher throughput. 2228 222911.1 High Availability in a Single Switch Topology 2230-------------------------------------------------- 2231 2232If two hosts (or a host and a single switch) are directly 2233connected via multiple physical links, then there is no availability 2234penalty to optimizing for maximum bandwidth. In this case, there is 2235only one switch (or peer), so if it fails, there is no alternative 2236access to fail over to. Additionally, the bonding load balance modes 2237support link monitoring of their members, so if individual links fail, 2238the load will be rebalanced across the remaining devices. 2239 2240See Section 12, "Configuring Bonding for Maximum Throughput" 2241for information on configuring bonding with one peer device. 2242 224311.2 High Availability in a Multiple Switch Topology 2244---------------------------------------------------- 2245 2246With multiple switches, the configuration of bonding and the 2247network changes dramatically. In multiple switch topologies, there is 2248a trade off between network availability and usable bandwidth. 2249 2250Below is a sample network, configured to maximize the 2251availability of the network:: 2252 2253 | | 2254 |port3 port3| 2255 +-----+----+ +-----+----+ 2256 | |port2 ISL port2| | 2257 | switch A +--------------------------+ switch B | 2258 | | | | 2259 +-----+----+ +-----++---+ 2260 |port1 port1| 2261 | +-------+ | 2262 +-------------+ host1 +---------------+ 2263 eth0 +-------+ eth1 2264 2265In this configuration, there is a link between the two 2266switches (ISL, or inter switch link), and multiple ports connecting to 2267the outside world ("port3" on each switch). There is no technical 2268reason that this could not be extended to a third switch. 2269 227011.2.1 HA Bonding Mode Selection for Multiple Switch Topology 2271------------------------------------------------------------- 2272 2273In a topology such as the example above, the active-backup and 2274broadcast modes are the only useful bonding modes when optimizing for 2275availability; the other modes require all links to terminate on the 2276same peer for them to behave rationally. 2277 2278active-backup: 2279 This is generally the preferred mode, particularly if 2280 the switches have an ISL and play together well. If the 2281 network configuration is such that one switch is specifically 2282 a backup switch (e.g., has lower capacity, higher cost, etc), 2283 then the primary option can be used to insure that the 2284 preferred link is always used when it is available. 2285 2286broadcast: 2287 This mode is really a special purpose mode, and is suitable 2288 only for very specific needs. For example, if the two 2289 switches are not connected (no ISL), and the networks beyond 2290 them are totally independent. In this case, if it is 2291 necessary for some specific one-way traffic to reach both 2292 independent networks, then the broadcast mode may be suitable. 2293 229411.2.2 HA Link Monitoring Selection for Multiple Switch Topology 2295---------------------------------------------------------------- 2296 2297The choice of link monitoring ultimately depends upon your 2298switch. If the switch can reliably fail ports in response to other 2299failures, then either the MII or ARP monitors should work. For 2300example, in the above example, if the "port3" link fails at the remote 2301end, the MII monitor has no direct means to detect this. The ARP 2302monitor could be configured with a target at the remote end of port3, 2303thus detecting that failure without switch support. 2304 2305In general, however, in a multiple switch topology, the ARP 2306monitor can provide a higher level of reliability in detecting end to 2307end connectivity failures (which may be caused by the failure of any 2308individual component to pass traffic for any reason). Additionally, 2309the ARP monitor should be configured with multiple targets (at least 2310one for each switch in the network). This will insure that, 2311regardless of which switch is active, the ARP monitor has a suitable 2312target to query. 2313 2314Note, also, that of late many switches now support a functionality 2315generally referred to as "trunk failover." This is a feature of the 2316switch that causes the link state of a particular switch port to be set 2317down (or up) when the state of another switch port goes down (or up). 2318Its purpose is to propagate link failures from logically "exterior" ports 2319to the logically "interior" ports that bonding is able to monitor via 2320miimon. Availability and configuration for trunk failover varies by 2321switch, but this can be a viable alternative to the ARP monitor when using 2322suitable switches. 2323 232412. Configuring Bonding for Maximum Throughput 2325============================================== 2326 232712.1 Maximizing Throughput in a Single Switch Topology 2328------------------------------------------------------ 2329 2330In a single switch configuration, the best method to maximize 2331throughput depends upon the application and network environment. The 2332various load balancing modes each have strengths and weaknesses in 2333different environments, as detailed below. 2334 2335For this discussion, we will break down the topologies into 2336two categories. Depending upon the destination of most traffic, we 2337categorize them into either "gatewayed" or "local" configurations. 2338 2339In a gatewayed configuration, the "switch" is acting primarily 2340as a router, and the majority of traffic passes through this router to 2341other networks. An example would be the following:: 2342 2343 2344 +----------+ +----------+ 2345 | |eth0 port1| | to other networks 2346 | Host A +---------------------+ router +-------------------> 2347 | +---------------------+ | Hosts B and C are out 2348 | |eth1 port2| | here somewhere 2349 +----------+ +----------+ 2350 2351The router may be a dedicated router device, or another host 2352acting as a gateway. For our discussion, the important point is that 2353the majority of traffic from Host A will pass through the router to 2354some other network before reaching its final destination. 2355 2356In a gatewayed network configuration, although Host A may 2357communicate with many other systems, all of its traffic will be sent 2358and received via one other peer on the local network, the router. 2359 2360Note that the case of two systems connected directly via 2361multiple physical links is, for purposes of configuring bonding, the 2362same as a gatewayed configuration. In that case, it happens that all 2363traffic is destined for the "gateway" itself, not some other network 2364beyond the gateway. 2365 2366In a local configuration, the "switch" is acting primarily as 2367a switch, and the majority of traffic passes through this switch to 2368reach other stations on the same network. An example would be the 2369following:: 2370 2371 +----------+ +----------+ +--------+ 2372 | |eth0 port1| +-------+ Host B | 2373 | Host A +------------+ switch |port3 +--------+ 2374 | +------------+ | +--------+ 2375 | |eth1 port2| +------------------+ Host C | 2376 +----------+ +----------+port4 +--------+ 2377 2378 2379Again, the switch may be a dedicated switch device, or another 2380host acting as a gateway. For our discussion, the important point is 2381that the majority of traffic from Host A is destined for other hosts 2382on the same local network (Hosts B and C in the above example). 2383 2384In summary, in a gatewayed configuration, traffic to and from 2385the bonded device will be to the same MAC level peer on the network 2386(the gateway itself, i.e., the router), regardless of its final 2387destination. In a local configuration, traffic flows directly to and 2388from the final destinations, thus, each destination (Host B, Host C) 2389will be addressed directly by their individual MAC addresses. 2390 2391This distinction between a gatewayed and a local network 2392configuration is important because many of the load balancing modes 2393available use the MAC addresses of the local network source and 2394destination to make load balancing decisions. The behavior of each 2395mode is described below. 2396 2397 239812.1.1 MT Bonding Mode Selection for Single Switch Topology 2399----------------------------------------------------------- 2400 2401This configuration is the easiest to set up and to understand, 2402although you will have to decide which bonding mode best suits your 2403needs. The trade offs for each mode are detailed below: 2404 2405balance-rr: 2406 This mode is the only mode that will permit a single 2407 TCP/IP connection to stripe traffic across multiple 2408 interfaces. It is therefore the only mode that will allow a 2409 single TCP/IP stream to utilize more than one interface's 2410 worth of throughput. This comes at a cost, however: the 2411 striping generally results in peer systems receiving packets out 2412 of order, causing TCP/IP's congestion control system to kick 2413 in, often by retransmitting segments. 2414 2415 It is possible to adjust TCP/IP's congestion limits by 2416 altering the net.ipv4.tcp_reordering sysctl parameter. The 2417 usual default value is 3. But keep in mind TCP stack is able 2418 to automatically increase this when it detects reorders. 2419 2420 Note that the fraction of packets that will be delivered out of 2421 order is highly variable, and is unlikely to be zero. The level 2422 of reordering depends upon a variety of factors, including the 2423 networking interfaces, the switch, and the topology of the 2424 configuration. Speaking in general terms, higher speed network 2425 cards produce more reordering (due to factors such as packet 2426 coalescing), and a "many to many" topology will reorder at a 2427 higher rate than a "many slow to one fast" configuration. 2428 2429 Many switches do not support any modes that stripe traffic 2430 (instead choosing a port based upon IP or MAC level addresses); 2431 for those devices, traffic for a particular connection flowing 2432 through the switch to a balance-rr bond will not utilize greater 2433 than one interface's worth of bandwidth. 2434 2435 If you are utilizing protocols other than TCP/IP, UDP for 2436 example, and your application can tolerate out of order 2437 delivery, then this mode can allow for single stream datagram 2438 performance that scales near linearly as interfaces are added 2439 to the bond. 2440 2441 This mode requires the switch to have the appropriate ports 2442 configured for "etherchannel" or "trunking." 2443 2444active-backup: 2445 There is not much advantage in this network topology to 2446 the active-backup mode, as the inactive backup devices are all 2447 connected to the same peer as the primary. In this case, a 2448 load balancing mode (with link monitoring) will provide the 2449 same level of network availability, but with increased 2450 available bandwidth. On the plus side, active-backup mode 2451 does not require any configuration of the switch, so it may 2452 have value if the hardware available does not support any of 2453 the load balance modes. 2454 2455balance-xor: 2456 This mode will limit traffic such that packets destined 2457 for specific peers will always be sent over the same 2458 interface. Since the destination is determined by the MAC 2459 addresses involved, this mode works best in a "local" network 2460 configuration (as described above), with destinations all on 2461 the same local network. This mode is likely to be suboptimal 2462 if all your traffic is passed through a single router (i.e., a 2463 "gatewayed" network configuration, as described above). 2464 2465 As with balance-rr, the switch ports need to be configured for 2466 "etherchannel" or "trunking." 2467 2468broadcast: 2469 Like active-backup, there is not much advantage to this 2470 mode in this type of network topology. 2471 2472802.3ad: 2473 This mode can be a good choice for this type of network 2474 topology. The 802.3ad mode is an IEEE standard, so all peers 2475 that implement 802.3ad should interoperate well. The 802.3ad 2476 protocol includes automatic configuration of the aggregates, 2477 so minimal manual configuration of the switch is needed 2478 (typically only to designate that some set of devices is 2479 available for 802.3ad). The 802.3ad standard also mandates 2480 that frames be delivered in order (within certain limits), so 2481 in general single connections will not see misordering of 2482 packets. The 802.3ad mode does have some drawbacks: the 2483 standard mandates that all devices in the aggregate operate at 2484 the same speed and duplex. Also, as with all bonding load 2485 balance modes other than balance-rr, no single connection will 2486 be able to utilize more than a single interface's worth of 2487 bandwidth. 2488 2489 Additionally, the linux bonding 802.3ad implementation 2490 distributes traffic by peer (using an XOR of MAC addresses 2491 and packet type ID), so in a "gatewayed" configuration, all 2492 outgoing traffic will generally use the same device. Incoming 2493 traffic may also end up on a single device, but that is 2494 dependent upon the balancing policy of the peer's 802.3ad 2495 implementation. In a "local" configuration, traffic will be 2496 distributed across the devices in the bond. 2497 2498 Finally, the 802.3ad mode mandates the use of the MII monitor, 2499 therefore, the ARP monitor is not available in this mode. 2500 2501balance-tlb: 2502 The balance-tlb mode balances outgoing traffic by peer. 2503 Since the balancing is done according to MAC address, in a 2504 "gatewayed" configuration (as described above), this mode will 2505 send all traffic across a single device. However, in a 2506 "local" network configuration, this mode balances multiple 2507 local network peers across devices in a vaguely intelligent 2508 manner (not a simple XOR as in balance-xor or 802.3ad mode), 2509 so that mathematically unlucky MAC addresses (i.e., ones that 2510 XOR to the same value) will not all "bunch up" on a single 2511 interface. 2512 2513 Unlike 802.3ad, interfaces may be of differing speeds, and no 2514 special switch configuration is required. On the down side, 2515 in this mode all incoming traffic arrives over a single 2516 interface, this mode requires certain ethtool support in the 2517 network device driver of the slave interfaces, and the ARP 2518 monitor is not available. 2519 2520balance-alb: 2521 This mode is everything that balance-tlb is, and more. 2522 It has all of the features (and restrictions) of balance-tlb, 2523 and will also balance incoming traffic from local network 2524 peers (as described in the Bonding Module Options section, 2525 above). 2526 2527 The only additional down side to this mode is that the network 2528 device driver must support changing the hardware address while 2529 the device is open. 2530 253112.1.2 MT Link Monitoring for Single Switch Topology 2532---------------------------------------------------- 2533 2534The choice of link monitoring may largely depend upon which 2535mode you choose to use. The more advanced load balancing modes do not 2536support the use of the ARP monitor, and are thus restricted to using 2537the MII monitor (which does not provide as high a level of end to end 2538assurance as the ARP monitor). 2539 254012.2 Maximum Throughput in a Multiple Switch Topology 2541----------------------------------------------------- 2542 2543Multiple switches may be utilized to optimize for throughput 2544when they are configured in parallel as part of an isolated network 2545between two or more systems, for example:: 2546 2547 +-----------+ 2548 | Host A | 2549 +-+---+---+-+ 2550 | | | 2551 +--------+ | +---------+ 2552 | | | 2553 +------+---+ +-----+----+ +-----+----+ 2554 | Switch A | | Switch B | | Switch C | 2555 +------+---+ +-----+----+ +-----+----+ 2556 | | | 2557 +--------+ | +---------+ 2558 | | | 2559 +-+---+---+-+ 2560 | Host B | 2561 +-----------+ 2562 2563In this configuration, the switches are isolated from one 2564another. One reason to employ a topology such as this is for an 2565isolated network with many hosts (a cluster configured for high 2566performance, for example), using multiple smaller switches can be more 2567cost effective than a single larger switch, e.g., on a network with 24 2568hosts, three 24 port switches can be significantly less expensive than 2569a single 72 port switch. 2570 2571If access beyond the network is required, an individual host 2572can be equipped with an additional network device connected to an 2573external network; this host then additionally acts as a gateway. 2574 257512.2.1 MT Bonding Mode Selection for Multiple Switch Topology 2576------------------------------------------------------------- 2577 2578In actual practice, the bonding mode typically employed in 2579configurations of this type is balance-rr. Historically, in this 2580network configuration, the usual caveats about out of order packet 2581delivery are mitigated by the use of network adapters that do not do 2582any kind of packet coalescing (via the use of NAPI, or because the 2583device itself does not generate interrupts until some number of 2584packets has arrived). When employed in this fashion, the balance-rr 2585mode allows individual connections between two hosts to effectively 2586utilize greater than one interface's bandwidth. 2587 258812.2.2 MT Link Monitoring for Multiple Switch Topology 2589------------------------------------------------------ 2590 2591Again, in actual practice, the MII monitor is most often used 2592in this configuration, as performance is given preference over 2593availability. The ARP monitor will function in this topology, but its 2594advantages over the MII monitor are mitigated by the volume of probes 2595needed as the number of systems involved grows (remember that each 2596host in the network is configured with bonding). 2597 259813. Switch Behavior Issues 2599========================== 2600 260113.1 Link Establishment and Failover Delays 2602------------------------------------------- 2603 2604Some switches exhibit undesirable behavior with regard to the 2605timing of link up and down reporting by the switch. 2606 2607First, when a link comes up, some switches may indicate that 2608the link is up (carrier available), but not pass traffic over the 2609interface for some period of time. This delay is typically due to 2610some type of autonegotiation or routing protocol, but may also occur 2611during switch initialization (e.g., during recovery after a switch 2612failure). If you find this to be a problem, specify an appropriate 2613value to the updelay bonding module option to delay the use of the 2614relevant interface(s). 2615 2616Second, some switches may "bounce" the link state one or more 2617times while a link is changing state. This occurs most commonly while 2618the switch is initializing. Again, an appropriate updelay value may 2619help. 2620 2621Note that when a bonding interface has no active links, the 2622driver will immediately reuse the first link that goes up, even if the 2623updelay parameter has been specified (the updelay is ignored in this 2624case). If there are slave interfaces waiting for the updelay timeout 2625to expire, the interface that first went into that state will be 2626immediately reused. This reduces down time of the network if the 2627value of updelay has been overestimated, and since this occurs only in 2628cases with no connectivity, there is no additional penalty for 2629ignoring the updelay. 2630 2631In addition to the concerns about switch timings, if your 2632switches take a long time to go into backup mode, it may be desirable 2633to not activate a backup interface immediately after a link goes down. 2634Failover may be delayed via the downdelay bonding module option. 2635 263613.2 Duplicated Incoming Packets 2637-------------------------------- 2638 2639NOTE: Starting with version 3.0.2, the bonding driver has logic to 2640suppress duplicate packets, which should largely eliminate this problem. 2641The following description is kept for reference. 2642 2643It is not uncommon to observe a short burst of duplicated 2644traffic when the bonding device is first used, or after it has been 2645idle for some period of time. This is most easily observed by issuing 2646a "ping" to some other host on the network, and noticing that the 2647output from ping flags duplicates (typically one per slave). 2648 2649For example, on a bond in active-backup mode with five slaves 2650all connected to one switch, the output may appear as follows:: 2651 2652 # ping -n 10.0.4.2 2653 PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data. 2654 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms 2655 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 2656 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 2657 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 2658 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 2659 64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms 2660 64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms 2661 64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms 2662 2663This is not due to an error in the bonding driver, rather, it 2664is a side effect of how many switches update their MAC forwarding 2665tables. Initially, the switch does not associate the MAC address in 2666the packet with a particular switch port, and so it may send the 2667traffic to all ports until its MAC forwarding table is updated. Since 2668the interfaces attached to the bond may occupy multiple ports on a 2669single switch, when the switch (temporarily) floods the traffic to all 2670ports, the bond device receives multiple copies of the same packet 2671(one per slave device). 2672 2673The duplicated packet behavior is switch dependent, some 2674switches exhibit this, and some do not. On switches that display this 2675behavior, it can be induced by clearing the MAC forwarding table (on 2676most Cisco switches, the privileged command "clear mac address-table 2677dynamic" will accomplish this). 2678 267914. Hardware Specific Considerations 2680==================================== 2681 2682This section contains additional information for configuring 2683bonding on specific hardware platforms, or for interfacing bonding 2684with particular switches or other devices. 2685 268614.1 IBM BladeCenter 2687-------------------- 2688 2689This applies to the JS20 and similar systems. 2690 2691On the JS20 blades, the bonding driver supports only 2692balance-rr, active-backup, balance-tlb and balance-alb modes. This is 2693largely due to the network topology inside the BladeCenter, detailed 2694below. 2695 2696JS20 network adapter information 2697-------------------------------- 2698 2699All JS20s come with two Broadcom Gigabit Ethernet ports 2700integrated on the planar (that's "motherboard" in IBM-speak). In the 2701BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to 2702I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2. 2703An add-on Broadcom daughter card can be installed on a JS20 to provide 2704two more Gigabit Ethernet ports. These ports, eth2 and eth3, are 2705wired to I/O Modules 3 and 4, respectively. 2706 2707Each I/O Module may contain either a switch or a passthrough 2708module (which allows ports to be directly connected to an external 2709switch). Some bonding modes require a specific BladeCenter internal 2710network topology in order to function; these are detailed below. 2711 2712Additional BladeCenter-specific networking information can be 2713found in two IBM Redbooks (www.ibm.com/redbooks): 2714 2715- "IBM eServer BladeCenter Networking Options" 2716- "IBM eServer BladeCenter Layer 2-7 Network Switching" 2717 2718BladeCenter networking configuration 2719------------------------------------ 2720 2721Because a BladeCenter can be configured in a very large number 2722of ways, this discussion will be confined to describing basic 2723configurations. 2724 2725Normally, Ethernet Switch Modules (ESMs) are used in I/O 2726modules 1 and 2. In this configuration, the eth0 and eth1 ports of a 2727JS20 will be connected to different internal switches (in the 2728respective I/O modules). 2729 2730A passthrough module (OPM or CPM, optical or copper, 2731passthrough module) connects the I/O module directly to an external 2732switch. By using PMs in I/O module #1 and #2, the eth0 and eth1 2733interfaces of a JS20 can be redirected to the outside world and 2734connected to a common external switch. 2735 2736Depending upon the mix of ESMs and PMs, the network will 2737appear to bonding as either a single switch topology (all PMs) or as a 2738multiple switch topology (one or more ESMs, zero or more PMs). It is 2739also possible to connect ESMs together, resulting in a configuration 2740much like the example in "High Availability in a Multiple Switch 2741Topology," above. 2742 2743Requirements for specific modes 2744------------------------------- 2745 2746The balance-rr mode requires the use of passthrough modules 2747for devices in the bond, all connected to an common external switch. 2748That switch must be configured for "etherchannel" or "trunking" on the 2749appropriate ports, as is usual for balance-rr. 2750 2751The balance-alb and balance-tlb modes will function with 2752either switch modules or passthrough modules (or a mix). The only 2753specific requirement for these modes is that all network interfaces 2754must be able to reach all destinations for traffic sent over the 2755bonding device (i.e., the network must converge at some point outside 2756the BladeCenter). 2757 2758The active-backup mode has no additional requirements. 2759 2760Link monitoring issues 2761---------------------- 2762 2763When an Ethernet Switch Module is in place, only the ARP 2764monitor will reliably detect link loss to an external switch. This is 2765nothing unusual, but examination of the BladeCenter cabinet would 2766suggest that the "external" network ports are the ethernet ports for 2767the system, when it fact there is a switch between these "external" 2768ports and the devices on the JS20 system itself. The MII monitor is 2769only able to detect link failures between the ESM and the JS20 system. 2770 2771When a passthrough module is in place, the MII monitor does 2772detect failures to the "external" port, which is then directly 2773connected to the JS20 system. 2774 2775Other concerns 2776-------------- 2777 2778The Serial Over LAN (SoL) link is established over the primary 2779ethernet (eth0) only, therefore, any loss of link to eth0 will result 2780in losing your SoL connection. It will not fail over with other 2781network traffic, as the SoL system is beyond the control of the 2782bonding driver. 2783 2784It may be desirable to disable spanning tree on the switch 2785(either the internal Ethernet Switch Module, or an external switch) to 2786avoid fail-over delay issues when using bonding. 2787 2788 278915. Frequently Asked Questions 2790============================== 2791 27921. Is it SMP safe? 2793------------------- 2794 2795Yes. The old 2.0.xx channel bonding patch was not SMP safe. 2796The new driver was designed to be SMP safe from the start. 2797 27982. What type of cards will work with it? 2799----------------------------------------- 2800 2801Any Ethernet type cards (you can even mix cards - a Intel 2802EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes, 2803devices need not be of the same speed. 2804 2805Starting with version 3.2.1, bonding also supports Infiniband 2806slaves in active-backup mode. 2807 28083. How many bonding devices can I have? 2809---------------------------------------- 2810 2811There is no limit. 2812 28134. How many slaves can a bonding device have? 2814---------------------------------------------- 2815 2816This is limited only by the number of network interfaces Linux 2817supports and/or the number of network cards you can place in your 2818system. 2819 28205. What happens when a slave link dies? 2821---------------------------------------- 2822 2823If link monitoring is enabled, then the failing device will be 2824disabled. The active-backup mode will fail over to a backup link, and 2825other modes will ignore the failed link. The link will continue to be 2826monitored, and should it recover, it will rejoin the bond (in whatever 2827manner is appropriate for the mode). See the sections on High 2828Availability and the documentation for each mode for additional 2829information. 2830 2831Link monitoring can be enabled via either the miimon or 2832arp_interval parameters (described in the module parameters section, 2833above). In general, miimon monitors the carrier state as sensed by 2834the underlying network device, and the arp monitor (arp_interval) 2835monitors connectivity to another host on the local network. 2836 2837If no link monitoring is configured, the bonding driver will 2838be unable to detect link failures, and will assume that all links are 2839always available. This will likely result in lost packets, and a 2840resulting degradation of performance. The precise performance loss 2841depends upon the bonding mode and network configuration. 2842 28436. Can bonding be used for High Availability? 2844---------------------------------------------- 2845 2846Yes. See the section on High Availability for details. 2847 28487. Which switches/systems does it work with? 2849--------------------------------------------- 2850 2851The full answer to this depends upon the desired mode. 2852 2853In the basic balance modes (balance-rr and balance-xor), it 2854works with any system that supports etherchannel (also called 2855trunking). Most managed switches currently available have such 2856support, and many unmanaged switches as well. 2857 2858The advanced balance modes (balance-tlb and balance-alb) do 2859not have special switch requirements, but do need device drivers that 2860support specific features (described in the appropriate section under 2861module parameters, above). 2862 2863In 802.3ad mode, it works with systems that support IEEE 2864802.3ad Dynamic Link Aggregation. Most managed and many unmanaged 2865switches currently available support 802.3ad. 2866 2867The active-backup mode should work with any Layer-II switch. 2868 28698. Where does a bonding device get its MAC address from? 2870--------------------------------------------------------- 2871 2872When using slave devices that have fixed MAC addresses, or when 2873the fail_over_mac option is enabled, the bonding device's MAC address is 2874the MAC address of the active slave. 2875 2876For other configurations, if not explicitly configured (with 2877ifconfig or ip link), the MAC address of the bonding device is taken from 2878its first slave device. This MAC address is then passed to all following 2879slaves and remains persistent (even if the first slave is removed) until 2880the bonding device is brought down or reconfigured. 2881 2882If you wish to change the MAC address, you can set it with 2883ifconfig or ip link:: 2884 2885 # ifconfig bond0 hw ether 00:11:22:33:44:55 2886 2887 # ip link set bond0 address 66:77:88:99:aa:bb 2888 2889The MAC address can be also changed by bringing down/up the 2890device and then changing its slaves (or their order):: 2891 2892 # ifconfig bond0 down ; modprobe -r bonding 2893 # ifconfig bond0 .... up 2894 # ifenslave bond0 eth... 2895 2896This method will automatically take the address from the next 2897slave that is added. 2898 2899To restore your slaves' MAC addresses, you need to detach them 2900from the bond (``ifenslave -d bond0 eth0``). The bonding driver will 2901then restore the MAC addresses that the slaves had before they were 2902enslaved. 2903 290416. Resources and Links 2905======================= 2906 2907The latest version of the bonding driver can be found in the latest 2908version of the linux kernel, found on http://kernel.org 2909 2910The latest version of this document can be found in the latest kernel 2911source (named Documentation/networking/bonding.rst). 2912 2913Discussions regarding the development of the bonding driver take place 2914on the main Linux network mailing list, hosted at vger.kernel.org. The list 2915address is: 2916 2917netdev@vger.kernel.org 2918 2919The administrative interface (to subscribe or unsubscribe) can 2920be found at: 2921 2922http://vger.kernel.org/vger-lists.html#netdev 2923