IP Address Allocation for AWS

With cloud computing many new techniques and technologies have penetrated into the IT landscape. Container orchestration is one such feature which accelerates the ability to quickly spin up new containers on a single node without the overhead of allocation and request for IP address.  An internal docket IP is assigned to containers and instances receive an IP address from their subnet and address using overlay. 

In today’s topic we will learn about how IP address allocation is done in AWS, what are its benefits and usage.

IP Address Allocation in AWS 

AWS introduced EKS CNI plugin to ensure compatibility with other network services such as VPC flow logs etc. and this led to assignment of IP addresses from VPC to individual nodes/pods as well. Figure1 below depicts the assignment.

Example Scenario

Let’s take an example to explain the scenario, if you have a cluster having 5 nodes and 10 pods. Adding a single demon set to all nodes would lead to consumption of 20 IP addresses. With a large IP address range also, this would quickly become the design limitation. To overcome this problem AWS VPC supports secondary IPv4 CIDR blocks (100.64.0.0/10 and 198.19.0.0/16) to open millions of additional IP addresses for pods / nodes for usage. There is a pod limit per node for each type to have control over the maximum number of pods / per instance type. 

• Using SNAT allows nodes in the public subnet to communicate with the public Internet, it translates network address into Internet gateway IP address. 

• NAT gateway is required by Nodes and pods running in private subnets.

• There are implications on choice of IP address range sufficient enough to host all instances and pods to address future requirements based on instance types as they determine how IP addresses are being used and allocated. 

• Once the instance type is determined you can now decide the number of instances required to host a number of business applications on cloud. 

• AWS CNI does allocation of the number of IP addresses it requires and keeps 1 spare ENI with available IP address. 

• A new EC2 instance will have allocated IP addresses from start and will create further ENI and allocate IP addresses for future usage. 

• All running pods will be assigned the IP address from first ENI, the second ENI will only partially assign IP address to pods which are running and third ENI is fully reserved as spare ENI. 

• AWS CNI plugin assigns IP addresses on nodes and performs configuration settings required on nodes. 

• By default, in AWS, based on the instance type chosen, a large pool of IP addresses are available on AWS EKS worker nodes also known as WARM pool.

• The free IP addresses required by L-IPAM daemon for allocation and available for pod assignment to nodes are mentioned in the WARM_IP_TARGET parameter property. 

• The LPAM daemon allocates additional secondary IP addresses to have a free pool within the same ENI. 

• Once the IP address allocation limit is reached with ENI, L-PAM daemon will attach new ENI and allocate new IP addresses. 

• When pods are no longer available, IP addresses and ENIs are released back to the VPC subnet.

• Use below command to allocate IP addresses to L-IPAM daemon 

kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=4

The WARM_IP_TARGET is very useful when more efficient and optimal IP address allocation is required without wasting IP addresses. This setting helps in moving away from default behaviour of AWS CNI and only allocate IP addresses as close as possible to required by pods. However, this feature is not so attractive as it  requires self-management of aws-node component post cluster creation and involves manual effort.