Technical Blueprint — Low-Latency SaaS Infrastructure

In this article I am outlining the technical difficulties that lay in front of anyone trying to solve the problem of low-latency distributed SaaS applications. And how those difficulties can be solved to an extent.

Success criteria are stateful applications, with the lowest possible latency shared between a group of users who are geographically close to each other. Application must be hosted by the SaaS provider; For two reasons — validation of computations by provider and high availability for all participants.

As you can imagine, this problem is being resolved at the moment in a few ways: by the use of cloud providers for computation in a few densely populated areas, Content Delivery Networks to send the connection to more remote locations, and the use of on-premises locations that cloud providers do not cover.

Yet, connecting and managing such a system together is a tough operation that only the largest companies like Valve take upon themselves. That might change soon and let me walk you through how.

The hard limitation for the audience of low-latency applications is physical networks. And while CDNs are great for providing connections between many different providers. The audiences are still vastly on their own — many hops of traffic add latency and disconnect audiences from high-quality experiences. That is a hard limitation — for some geographical regions, only new infrastructure projects on the ground and under the sea can solve those issues. Breakthrough technologies like Starlink provide connectivity in far more remote areas, yet their latency is still prohibitive for low-latency applications.

Soft limitation is a task — interconnecting traffic between different Cloud Providers, CDNs, and hardware. This limitation has already been overcome and and readily available.

To overcome the hard limitation of network speed and connectivity, we will have to provision hosting of SaaS applications as close as possible to its users. That comes with a cost and a question of how to control such infrastructure.

One-size-fits-all would be a Kubernetes cluster with nodes across many locations. According to these benchmarks, Containers ran just 0.12% slower than native processes.

To provision nodes on-demand you would have to operate an infrastructure solution like Terraform, to trigger APIs of Cloud Providers (AWS, GCP, Azure), data centers using VMware, and other hypervisors. A big chunk of work would be dedicated to maintaining this solution.

The progress that Cloud Providers made and is now taken for granted is the Pay-As-You-Go model. You can start using a virtual machine on AWS as soon as you attach your credit card. No strings attached.

Yet many of the classic data centers are still picking up the pace for provisioning their environment online via private cloud tools. One of the big thresholds would be the performance of those machines and measuring bang-per-buck. Yet running a pilot is always best to get a benchmark specific to your workload.