Are you encountering problems running more than a few tasks per ECS host? If so, read on and learn about a simple fix that will drastically increase the number of ECS tasks you can run.
Background
The default configuration of ECS on EC2 restricts the number of ECS tasks that can run on each node. The exact limit depends on the type of instance being used. This limitation exists because each task requires an Elastic Network Interface (ENI) to be assigned, which restricts the maximum number of tasks per instance.
ENI trunking to the rescue!
ENI trunking (with the awsvpcTrunking account setting) allows a host to run many more tasks per host. This is achieved using the primary ENI for host communication and additional ENI’s are assigned for trunking. Additional ENI’s are added as branch interfaces up to the maximum number for that instance type.
Let’s take a pretty common node type of r5.xlarge. By default, it supports three tasks, but if ENI trunking is enabled, that number goes up to 20. ( Reference table of Maximum Tasks per Instance type.)
Enabling trunking for an existing cluster
Enabling trunking when creating an ECS cluster works well, but what if you have an existing cluster and now want to enable trunking?
First, check to see if trunking is enabled with the below command. You can execute it in CloudShell or an appropriately configured CLI:
aws ecs list-attributes --target-type container-instance --attribute-name ecs.awsvpc-trunk-id --cluster <Cluster Name> --region <Region>
If an empty attributes list is returned, the specified cluster doesn’t have trunking enabled.
Trunking Considerations
Before enabling trunking, it's worth checking the AWS Documentation, which outlines its limitations.
The most common considerations with ENI trunking are:
- Only awsvpc network mode is supported.
- Launch type must be EC2 (Fargate does not need this feature, dedicated ENI is provided for each task by default).
- Shared networks are not supported.
- Windows containers are not supported.
- Service linked role must exist (created by default when ECS cluster is created).
- Resource-based IPV4 (A record) DNS requests must not be enabled (default when EC2 instances are created).
Let’s enable trunking
If you use the AWS Web Console to enable trunking on an existing cluster, the interface will show it’s enabled, but you’re likely to find the aws ecs list-attributes command returns an empty attributes array, which indicates it’s actually not enabled.
AWS Console showing, possibly incorrectly, that trunking is enabled
If that’s the case, you can enable trunking for all clusters in the account by executing the following command:
aws ecs put-account-setting-default --name awsvpcTrunking --value enabled --region <Region>
With this method, you only need to set the default once per region for a given account, as this command sets the default account setting property for trunking.
New ECS instances will pick up the configuration change, or you can restart already deployed instances. If using an autoscaling group, expand the number of instances deployed and recycle nodes to have the change take place without disruption.
You can verify that trunking is working by checking the tasks per host count or by rerunning the aws ecs list-attributes command. This command should now return a list of interfaces enabled for trunking rather than an empty attributes array.
If you don’t know DoiT yet you should definitely check us out. Here, our team is ready to learn more about you and your cloud engineering needs. Staffed exclusively with senior engineering talent, we specialise in providing advanced cloud consulting architectural design and debugging advice. Get in touch, and let’s chat!