Etherchannel Basics

Submitted by rayc on Mon, 10/25/2021 - 10:15

We know that in a fully redundant Layer 2 topology, STP will find a Root Bridge, Find the shortest path to the Root Bridge, and block all other ports as shown in the below diagram.

STP example topology

In the above topology, we have two links between SW2 and SW3 and one is effectively shutdown by STP. This seems like a bit of a wasted link though right? We have a link sitting there doing nothing that we could be utilising. This is where Etherchannels come in. An Etherchannel is a virtual grouping of physical interfaces so that logically, they act as one. Etherchannel or port-channel is referred to in the IEEE 802.3AD standard. 

When using etherchannels to bundle physical interfaces together, STP will see the port as a single logical interface, thus allowing traffic to flow across both links. Effectively increasing the bandwidth between switches. There are 3 configuration options for ether channels

  • On: This mode tells the switch to enable the port in an ether channel bundle. There is no negotiation. 
  • PAgP: This is a Cisco proprietary etherchannel protocol that negotiates an ether channel bundle and performs health checks to ensure links are active. 
  • LACP: This is the industry standard etherchannel protocol that negotiates an ether channel bundle and performs health checks to ensure links are active. 

 

So we've got multiple links connected to 2 switches and we've added them to an ether channel bundle. Great now we have 2Gbps throughput and the switch will evenly load balance across the two links right? While that is semi accurate, it's not quite how ether channels load balance. An switch does just receive a frame and go okay you go on port 1 of the bundle. The next frame goes out port 2 and so on. What happens is the switch looks at the frame and depending on the load balancing method thst you have configured, will calculate a hash value based on the frame information and go right, that frame gets sent out this link. The next frame gets assessed and a has created and depending on the hash value it might get sent over the same link or a different link. 

There are quite a few different load balancing configurations available on Cisco switches and they are platform dependent. In my lab here I have a 2960X and a 3650 series catalyst switch. These two particular models support the following load balancing algorithms:

3650

etherchannel load balancing options for 3650

2960X

etherchannel load balancing options for 2960X

As you can see the beefier 3650 running IOS-XEthat is capable of Layer 3 routing has far more load balancing options. The default load balancing option on both devices is src-mac.

default load balancing

With a load-balancing method of src-mac, the switch will look at each frame, calculate it's hash which will return one of 8 values (There can only be 8 ports in an etherchannel) and this value will determine which Etherchannel link the frame will be forwarded. From that point on, all frames coming from the same source MAC address, will be forwarded out the same Etherchannel port. Below is an example of how the hashing works with the number of ports. As I mentioned, the switch will assign 8 values to it's etherchannel port. The below table shows how many values each port will get.

Number of ports load balancing
8 1:1:1:1:1:1:1:1
7 2:1:1:1:1:1:1
6 2:2:1:1:1:1
5 2:2:2:1:1
4 2:2:2:2
3 3:3:2
2 4:4

As you can see in order to get an even distribution across all etherchannel links, it's better to use powers of 2. Given the number of ports and the load balancing algorithm of src-mac, you can understand why in some cases this would not result in optimal usage of links. For example if you have only one or two hosts that are sending the majority of the traffic, the load balancing algorithm will always be choosing the same physical interface. It is recommend that you find and use a load balancing algorithm that best suits your network to optimally load balance across all physical links. The best way to see the load of each interface in an etherchannel is to use the show command show etherchannel port.

show etherchannel port output

Cisco documentation mentions that if the load balancing method of dest mac, the switch is only capable of load balancing across 4 of the ports even if you have 8 in the etherchannel bundle. 

Etherchannel mode on

Because I will be explaining PAgP and LACP in other articles, in this article, I will only go through the Etherchannel mode On. In Etherchannel mode on, the port is always in an etherchannel state. There is no negotiation between the other end, and there is not active checking of the status of the device on the other end or the links. Using the topology in the above figure, I will enable etherchannel on the two links between SW2 and SW3.

When enabling a port in an etherchannel, there following port configuration must match on both ends of the link:

  • Port type: Every port must be either L2 switchport or L3 Routed
  • Port mode: All L2 etherchannel ports must be configured as either Access, or Trunk.
  • Native VLAN: The Native VLAN must match on Trunk ports
  • Allowed VLAN: The VLANs permitted on the Trunk port must match
  • Speed and Duplex: The Interface speed and duplex must match
  • MTU: All L3 interfaces MTU must match. 
  • Load Interval: The Load interface on all member ports must be the same
  • Storm Control: The Storm Control configuration on all member ports must be the same

 

To configure a switchport in etherchannel mode on, use the interface configuration subcommand channel-group <number> mode on. You can have up to 128 etherchannel groups on a Cisco switch.

Enabling on mode etherchannel on SW2

When you first configure a port in an etherchannel group, it creates a logical interface called a port-channel. Aside from a few specific commands like storm-control, all interface configuration for an etherchannel, should now be done under the port-channel <num> interface. Now let's configure the etherchannel port on SW3 and verify to etherchannel bundle status using the show command show etherchannel summary.

Enable etherchannel mode on on SW3 and view status

As you can see, both ports in the etherchannel bundle are in the down state. If we take a look at SW2 and SW3's show interface status output, we can see that SW2 has put the port-channel in an err-disable state and if we take a look at the output of the show interface status errdisable command, we can see that it is due to channel-missconfig. Let's take a look at the interface configuration on SW2.

show run interface miss configuration on SW2

As per the items listed above that must match in order to form an etherchannel, we can see that G1/0/3 is configured as a Trunk port, but G1/0/4 is not. After configuring G1/0/4 as a Trunk port, we can now see that the port-channel has come up and is working as expected. Remember, to clear an err-disable state you need to perform a shut no shut on the port-channel interface.

Fix Trunk config on SW2 and confirm etherchannel is up

Additional Port-Channel configuration

When enabling a port-channel, there are a few configuration options other than the load-balancing method that can be configured. You can configure the minimum number of required interfaces to be included in the bundle before the port channel can come up (This does not include if a link fails). To configure the minimum number of interfaces required, use the port-channel interface subconfiguration command port-channel min-links <number>. The number can be a range from 2-8. 

port-channel min-links command

If there are not enough interfaces in an UP state in the etherchannel, the status in the show etherchannel summary command output will be SM. S for layer 2, and M for min links not met.

Port-channel down due to min links not met

Layer 3 Port-Channel Configuration

The above configuration configures a layer 2 port, in a Layer 2 port-channel. To configure a Layer 3 port channel, you first would need to make the switch ports routed ports by using the interface subconfiguration command no switchport. Once the port is in routed mode, you can then add the port to the etherchannel group using the same command channel-group <num> mode <mode>.

Configuring a Layer 3 etherchannel

Spanning Tree

Just a quick note about spanning tree, now that we have the two links between SW2 and SW3 configured in an etherchannel, if we take a look at the output of the show spanning-tree command, we can see that we not longer see the individual ports but the single Port-Channel interface and the port cost has been adjusted as such as well. Also notice the port number.

show spanning-tree post etherchannel config