DoiT Cloud Intelligence™
Scaling Kubernetes: How to Seamlessly Expand Service IP Ranges
When managing Kubernetes clusters, ensuring sufficient IP addresses for Kubernetes Services can become a critical aspect of scaling and maintaining your infrastructure.
Services provide an abstract way to expose an application running on a set of Pods. Different types of Services, like ClusterIP, NodePort, and LoadBalancer, utilize a unique cluster-scoped virtual IP address called ClusterIP.
Each Service cluster IP address within the entire cluster must be unique. Attempting to create a Service with a ClusterIP that has already been allocated will result in an error. As your deployment expands, the default Service IP range may become inadequate, potentially leading to bottlenecks in network resources.
Previously, users could not resize or increase the ranges of IPs assigned to Services, causing problems when there were overlapping networks or the cluster ran out of available IPs.
However, kube-apiserver that have enabled theMultiCIDRServiceAllocator feature gate and the networking.k8s.io/v1alpha1 API allows users to dynamically expand the number of IPs available for Services. This feature is graduated to the beta stage in kubernetes 1.31 and it is recommended to wait for the feature to graduate to Stable before using it in production. Refer to KEP-1880—Multiple Service CIDRs for more details about the implementation.
In this blog post, we’ll explore how to extend the Service IP range in a Google Kubernetes Engine (GKE) cluster. You can also test the feature in other kubernetes distributions and refer to the release notes specific to the distributions for configuration or limitation details.
Prerequisites
You must have a GKE cluster with version 1.31.1-gke.1361000 or later, and ensure that the kubectl command-line tool is configured to communicate with your cluster.
Enable and Verify the Beta APIs
Starting with Kubernetes version 1.24, new beta APIs are disabled by default in new GKE clusters. Existing clusters created running a version earlier than 1.24 keep existing beta APIs enabled.
You can either enable the beta APIs during the cluster creation process or later. To enable the required beta APIs, follow the instructions in Use Kubernetes beta APIs with GKE clusters.
Sample command to enable the required beta APIs on an existing cluster:
gcloud container clusters update <<GKE_CLUSTER_NAME>> \
--enable-kubernetes-unstable-apis=networking.k8s.io/v1beta1/ipaddresses,networking.k8s.io/v1beta1/servicecidrs
--region <<GKE_CLUSTER_REGION>> \
--project <<GCP_PROJECT_NAME>>
The newly enabled APIs will create a ServiceCIDR object using a well-known name kubernetes and an IP address range based on the initial service CIDR.
Run the below command to verify the new ServiceCIDR object,
Chimbus-MacBook-Pro:~ chimbu$ kubectl get servicecidr
NAME CIDRS AGE
kubernetes 10.96.0.0/28 98m
Chimbus-MacBook-Pro:~ chimbu$
Adding a new ServiceCIDR
For testing purposes, I have created a cluster with /28 range for services which contains only 14 IP addresses. The kubernetes.default Service is always created; for this example, that leaves you with only 13 possible Services.
For testing purposes, I have created a cluster with a /28range for services that contains only 14 IP addresses. The kubernetes.default service is always created and that leaves you with just 13 possible Services.
Try creating additional kubernetes services and the request will fail with an internal error once all the ip addresses in the service range are utilized.
Chimbus-MacBook-Pro:~ chimbu$ for i in $(seq 1 13); do kubectl create service clusterip "service-test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done
10.96.0.13
10.96.0.11
10.96.0.7
10.96.0.14
10.96.0.3
10.96.0.12
10.96.0.9
10.96.0.4
10.96.0.5
error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full
error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full
error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full
error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full
Chimbus-MacBook-Pro:~ chimbu$
You can now expand the number of available IP addresses for Services by creating a new ServiceCIDR that extends or adds new IP address ranges.
During the beta phase, GKE allows you to only create Service CIDRs in the
34.118.224.0/20reserved IP address range to avoid possible issues with overlapping IP address ranges.
Apply the below manifest to create a new ServiceCIDR that adds a new IP address range.
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: ServiceCIDR
metadata:
name: newcidr1
spec:
cidrs:
- 34.118.224.0/20
EOF
Chimbus-MacBook-Pro:~ chimbu$ kubectl get servicecidrs.networking.k8s.io
NAME CIDRS AGE
kubernetes 10.96.0.0/28 104m
newcidr1 34.118.224.0/20 5s
Chimbus-MacBook-Pro:~ chimbu$
Try creating new kubernetes services and the cluster IP address will be picked from this new range.
Chimbus-MacBook-Pro:~ chimbu$ for i in $(seq 1 13); do kubectl create service clusterip "service-test-new-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done
34.118.235.89
34.118.234.162
34.118.230.252
34.118.226.209
34.118.227.183
34.118.227.182
^C
Chimbus-MacBook-Pro:~ chimbu$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 105m
service-test-1 ClusterIP 10.96.0.13 <none> 80/TCP 3m41s
service-test-2 ClusterIP 10.96.0.11 <none> 80/TCP 3m39s
service-test-3 ClusterIP 10.96.0.7 <none> 80/TCP 3m39s
service-test-4 ClusterIP 10.96.0.14 <none> 80/TCP 3m38s
service-test-5 ClusterIP 10.96.0.3 <none> 80/TCP 3m37s
service-test-6 ClusterIP 10.96.0.12 <none> 80/TCP 3m36s
service-test-7 ClusterIP 10.96.0.9 <none> 80/TCP 3m35s
service-test-8 ClusterIP 10.96.0.4 <none> 80/TCP 3m34s
service-test-9 ClusterIP 10.96.0.5 <none> 80/TCP 3m33s
service-test-new-1 ClusterIP 34.118.235.89 <none> 80/TCP 8s
service-test-new-2 ClusterIP 34.118.234.162 <none> 80/TCP 8s
service-test-new-3 ClusterIP 34.118.230.252 <none> 80/TCP 7s
service-test-new-4 ClusterIP 34.118.226.209 <none> 80/TCP 6s
service-test-new-5 ClusterIP 34.118.227.183 <none> 80/TCP 5s
service-test-new-6 ClusterIP 34.118.227.182 <none> 80/TCP 4s
Chimbus-MacBook-Pro:~ chimbu$
Deleting a ServiceCIDR
The ServiceCIDRs are protected with finalizers to avoid leaving Service ClusterIPs orphans; the finalizer is only removed if no IP addresses are assigned to any service belonging to the subnet.
So you need to first delete all the Kubernetes services containing the IP addresses of the ServiceCIDR and then delete the ServiceCIDR object.
In summary, as Kubernetes clusters expand, increasing the Service IP range becomes crucial to avoid IP exhaustion and facilitate smooth scalability. The new features offer flexibility for environments that require additional service IP addresses over time without risking resource bottlenecks.
I trust this blog post has provided valuable insights. If you’d like to know more or are interested in our services, don’t hesitate to get in touch. You can contact us here.