DoiT acquires SELECT to optimize Snowflake Waste
DoiT Cloud Intelligence™DoiT Cloud Intelligence™

DoiT Cloud Intelligence™

Upgrade with Confidence: Managing Deprecated APIs for Smooth GKE Transitions

By Felipe MartinezJun 13, 20244 min read
Upgrade with Confidence: Managing Deprecated APIs for Smooth GKE Transitions

Upgrading your GKE cluster can sometimes hit unexpected roadblocks. This blog post investigates a real-world scenario where a seemingly simple upgrade to version 1.27 was blocked by a “deprecated API call.”

GKE pauses automatic upgrades

If GKE detects usage of a deprecated feature or API, GKE pauses automatic upgrades to prevent your cluster from being upgraded into a broken state. Upgrades to the next Kubernetes minor version are paused, but GKE continues to deliver patch upgrades to the cluster on the current minor version.

Once GKE has not detected usage of the deprecated feature or calls to deprecated APIs for 30 days, the cluster is automatically upgraded if the next minor version is the default version in the cluster’s release channel.

If GKE continues to detect usage of the deprecated feature on the cluster, then GKE keeps the cluster on its current minor version until the version’s end of life date. Once this date — which is available in the Release Schedule — is reached, the cluster is automatically upgraded to the next minor version and the cluster environment might be impaired as the removed feature is still being used.

GKE Upgrade Blocked

On this case, GKE identified a call to a deprecated API as the reason for preventing the upgrade from version 1.26. Despite searching the cluster, no trace of the culprit — a “horizontalPodAutoscaler” using a deprecated API — could be found.

Investigative Steps

  • GKE Insights: While GKE Insights revealed the deprecated API call (horizontalPodAutoscaler v2beta2) and a user agent ("v2.3.0"), the user agent didn’t provide any meaningful information

  • Deprecated API: Looking into the 1.27 deprecated API list we can see the HPA v2beta2 will not be available on clusters running Kubernetes 1.26

  • Admin Activity Audit Logs: The first recommended way to investigate this is to locate API clients making write calls to the deprecated API with the log query below. The role roles/logging.viewer is required for this query.

resource.type="k8s_cluster"
labels."k8s.io/removed-release"="1.26"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system:")
  • Data Access Audit Logs : As next step let’s locate the API clients making read calls to deprecated APIs. We first need to enable Kubernetes data read and Admin read. You might need the role roles/logging.privateLogViewer as well so see the logs.

The Culprit Revealed

After waiting for the logs to populate, the culprit emerged: an outdated version ( 2.3.0) of kube-state-metrics! Kube-state-metrics, a monitoring service for Kubernetes objects, was making “ list” calls to the deprecated v2beta2 API, triggering the upgrade blocker. This case highlights how interconnected cloud environments can be.

Checking the source code you can find exactly where the call came from.

The Resolution

Upgrading kube-state-metrics to a version that doesn’t use the deprecated API resolved the issue and paved the way for a smooth GKE upgrade.

Conclusion

  • Logging bucket exclusion filter: Check the logging bucket exclusion filter to ensure the data access audit logs are not excluded to speed up the investigation.
  • Stay Informed About API Changes: Proactively monitor Kubernetes API changes to anticipate potential upgrade roadblocks.
  • Understand Static Versions vs. Release Channels: Weigh the trade-offs between static versions (stability) and release channels (automatic security fixes) for your upgrade strategy.
  • Manage Upgrades Strategically: Leverage auto-upgrade and maintenance exclusions to control upgrade timing, while considering allowing patch upgrades for critical security fixes.
  • Be Mindful of Log Costs: Enabling data access audit logs can incur additional costs. Evaluate your needs before enabling them for extended periods.

By following these tips, you can approach GKE upgrades with greater confidence and minimize disruptions to your cloud operations

If you don’t know DoiT International yet you should definitely check us out. Here, our team is ready to learn more about you and your cloud engineering needs. Staffed exclusively with senior engineering talent, we specialise in providing advanced cloud consulting architectural design and debugging advice. Get in touch, and let’s chat!

I hope you found this post informative! If you have any questions, please feel free to leave a comment below.