NIC Bonding

Network bonding is the aggregation or combination of multiple LAN cards into a single bonded interface to provide high availability and redundancy. Network bonding is also known as NIC teaming.

This pages shows how to bind multiple network interfaces into a single load balanced or fault-toleranced interface and so on.

Bonding mode

The Linux bonding driver allows system administrators to set up bonding interfaces with different modes. A bonding mode specifies the policy indicating how bonding slaves are used during network transmission. To achieve the maximum throughput and fault toleration, it is important to choose the proper bonding mode and the corresponding options for the setup

In Netplan, the default is balance-rr (round robin). Possible values are balance-rr, active-backup, balance-xor, broadcast, 802.3ad, balance-tlb, and balance-alb. For Open vSwitch active-backup and the additional modes balance-tcp and balance-slb are supported

Mode	Description	Load balancing	Fault tolerance
balance-rr or 0	Round-robin policy. Transmits packets in sequential order from the first available slave through the last.	O	O
active-backup or 1	Active-backup policy. Establishes that only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.	X	O
balance-xor or 2	Transmits based on the selected transmit hash policy, which can be altered via the xmit_hash_policy option	O	O
broadcast or 3	Transmits everything on all slave interfaces	X	O
802.3ad or 4	IEEE 802.3ad Dynamic link aggregation policy, This mode requires a switch that supports IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification	O	O
balance-tlb or 5	Adaptive transmit load balancing. Establishes channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.	O	O
balance-alb or 6	Adaptive load balancing. Includes balance-transmit load balancing plus receive-load balancing for IPv4 traffic, and does not require any special switch support. The receive-load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond. Thus, different peers use different hardware addresses for the server	O	O

Checking the bonding interface

$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 1
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: eno2np1
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:73:67:bb
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1

Slave Interface: eno1np0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:73:67:ba
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1

$ ethtool bond0
Settings for bond0:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Link detected: yes

Check LACP parameters

LACP parameters can also be verified from tcpdump on member interfaces

# tcpdump -i  ens1f0 -vv -s 1500
...
Actor Information TLV (0x01), length 20
          System 4c:16:fc:52:e4:00 (oui Unknown), System Priority 127, Key 10, Port 3, Port Priority 127
          State Flags [Activity, Timeout, Aggregation, Synchronization, Collecting, Distributing]
          0x0000:  007f 4c16 fc52 e400 000a 007f 0003 3f00
          0x0010:  0000
        Partner Information TLV (0x02), length 20
          System 90:e2:ba:cc:27:70 (oui Unknown), System Priority 65535, Key 15, Port 1, Port Priority 255
          State Flags [Activity, Timeout, Aggregation, Synchronization, Collecting, Distributing]
          0x0000:  ffff 90e2 bacc 2770 000f 00ff 0001 3f00
          0x0010:  0000
...