One technique to save your AWS EKS IP addresses 10x

Story

When I was doing a research to design AWS EKS clusters from the ground up. Networking aspect was a part of my considerations. One problem was that how many IP addresses I need in a long run. While it had no simple answer, instead I could tell you that how to save your IP addresses.

Prerequsites

The technique I am going to introduce is based on AWS CNI plugin (Amazon VPC Container Network Interface plugin for Kubernetes), which is also the default CNI of your EKS cluster.

From the offical documentation, Using this plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network. This is why your pods can communicate to computing resources outside your clusters, or even in other VPCs.

And let me roughly explain the real-world scenerio:

Each node group provisons a group of EC2 instances. Each instance has multiple ENIs, and one ENI contains multiple IP addresses. AWS CNI plugin helps you to manage these IP addresses and map to your EKS pods in a 1-to-1 relationship.

EKS features

Assign address prefixes to your ENI

By default, the number of IP addresses available to assign to pods is based on the number of IP addresses assigned to Elastic network interfaces, and the number of network interfaces attached to your Amazon EC2 node, i.e. AWS CNI plugin assigns IP addresses to ENIs.

EKS offers your another options, let AWS CNI plugin to assign / 28 IP address prefixes to ENIs.

Let me show you a simple example.

Consider that a ENI has ten slots:

• If you use the default mode, you can assign 10 IP addresses to it.
• If you use the prefixes mode, you can assign 10 /28 prefix blocks to it, i.e. 10 * 16 IP addresses = 160 IP addresses.

If you find out your instances have not fully utilized, you have a choice to optimize the pod-to-instance ratio and help to create fewer instances.

If it does not sound useful to you, prefixes mode has an addtional advantage. You can make less API calls to configure the network interfaces and IP addresses necessary for pod connectivity.

In the situation of massively scaling, API throttling might happen. Your EKS will stuck in a dead loop when it scales too quick.

Your pods need IP addresses, AWS VPC CNIS will call APIs to allocate IP addresses for your pods. However API throttling prevent you to allocate IP addresses, then AWS VPC CNI keeps sending API calls ....

In the default mode, you are assigning a single IP addresses each time. Now you can assign 16 IP addresses each time. While it is not 100% correct, you can roughly assuming that you can save 16x API calls.

CNI custom networking

By default, when new network interfaces are allocated for pods, AWS CNI plugin uses the security groups and subnet of the node's primary network interface, i.e. ** both pods and nodes are created in the same subnet.**

Normally, we use RFC 1918 CIDR ranges (10.0.0.0/8, 172.16.0.0/12 and 192.16.0.0/16) to allocate our IP addresses.
However, many of you might now know, EKS supports additional IPv4 CIDR blocks in the 100.64.0.0/10 and 198.19.0.0/16 ranges.

The trick is simple. You can create your nodes in RFC 1918 CIDR ranges and then create your pods in 100.64.0.0/10 and 198.19.0.0/16 ranges. You can much more IP addresses to allocate now :)

Please be reminded that 100.64.0.0/10 and 198.19.0.0/16 are not common IP ranges. Proper routing needs to be configured if you want the pods created in 100.64.0.0/16 and 192.19.0.0/16 to be able to communicate with RFC 1918 CIDR ranges.

Still if your pods do not need to communicate with peered networks (through VPC peering, transit gateway peering), you are safe to use these IP ranges.

Conclusion

Depends on your use cases, you can enable either or both Assign address prefixes to your ENI and CNI custom networking feature.

I also encourage you to understand what AWS VPC CNI plugin does if you do not aware of it before. It can help you to design EKS cluster in a better way.

If you are interested to use these features, AWS documentation has a very detailed explanation, which I have attached below. You should also checkout considerations and prerequisites in the documentation.

Reading

cni-increase-ip-addresses
cni-custom-network
Using additional IPv4 IP ranges
eni-max-pods