OpenSearch CCR across regions with custom packages requires redefining settings in the replication plugin API call.
Problem
An issue arises when you use a custom package in your index settings and replicate it to a domain in a different region. To associate a package to a domain, you must first upload the package to an S3 bucket in the same region as the cluster. Then you can import the package and associate it with the domain. When you do so, AWS automatically assigns a region-specific reference path. If you try to create a cross cluster connection and replicate, you’ll get a 502 error because the reference path for the package doesn’t exist in the follower domain.
Solution
When you create the leader index, you have to set the updateable flag to true. Then, when you start the replication process, you must redefine the settings with the follower reference path.
POC
Setup Prerequisites
- Leader Domain
- Follower Domain
- Connection between the leader and follower
- Custom packages in S3 buckets in each region
In addition, there are some requirements for the leader and follower domains as outlined by AWS [1]
As stated before, to associate a package with a domain, you must first upload the package to an S3 bucket in that domain’s region and then import it within OpenSearch under the packages option on the side bar.
Import Package
Here is a basic leader domain in US-east-1. AWS requirements include fine grained access control (FGAC) enabled and the security policy to allow the follower domain es* actions.
You can see under packages, I’ve associated a wordlist package which has the AWS-assigned reference path analyzers/F111929843.
Leader Domain
Here is the basic follower domain. The setup is identical to the leader domain except without the revised security policy. You can see under packages, I have associated the wordlist file here as well (after uploading it to the S3 path for that region), and the AWS-assigned reference path is analyzers/F194925348, which differs from the path for the leader domain.
To allow CCR, you must create a remote data connection request from the follower domain. This connection has to be then accepted in the leader domain.
You’re ready to replicate after the two domains with attached packages and data connection are in place.
Here is an example of setting up an index on the leader:
Your index settings might look something like this: note especially the filter path and the parameter updateable.
Create Index on Leader
Now when you start replication, you include the settings again and this time include the follower reference path.
Start Replication to Follower
To test out the replication, I added data to the leader as follows.
Adding Data to Index
Here is a search for some data in the follower. The query succeeded, proving the replication is working.
Query Data on the Follower
Results of Query on Follower
In conclusion, if you want to do CCR on AWS OpenSearch across regions with custom packages, you must redefine your settings when calling the replication plugin API.
Reference
[1]https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html
Reach out to us at DoiT . We specialize in providing advanced cloud consulting architectural design and consulting services to help organizations harness the cloud to drive business growth.