CodexMCP Automation: OpenSearch Cluster
OpenSearch is a core part of CodexMCP, acting as the primary data store and search engine. Unlike other components that can be installed with minimal configuration, OpenSearch required a fully customized setup from the beginning.
This was one of the most complex parts of the automation to get working due to its multiple dependencies, strict security controls, and the need for a custom synonym list for optimized search indexing. Below is a breakdown of each step in the OpenSearch installation and configuration process.
OpenSearch Installation
- "sudo apt-get update && sudo apt-get install -y lsb-release ca-certificates curl gnupg2"
This ensures the system has essential packages installed before adding the OpenSearch repository:
lsb-release
: Provides Linux distribution information.ca-certificates
: Ensures SSL certificates are up to date.curl
: Required for downloading external repositories.gnupg2
: Enables secure key handling for package verification.
- "curl -fsSL https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo gpg --dearmor -o /usr/share/keyrings/opensearch-keyring.gpg"
- Downloads and stores the OpenSearch GPG key, which verifies the authenticity of OpenSearch packages.
--dearmor
converts the ASCII-armored key to a format suitable for APT package verification.
- "echo \"deb [signed-by=/usr/share/keyrings/opensearch-keyring.gpg] https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main\" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list > /dev/null"
- Adds the OpenSearch APT repository to the system.
- This ensures OpenSearch can be installed and updated directly from OpenSearch’s official package servers.
- "sudo apt-get update"
- Updates the system’s package list to include OpenSearch.
- "sudo apt-get install -y opensearch"
- Installs OpenSearch from the newly added repository.
Fixing Permissions
Since OpenSearch runs as a dedicated opensearch
user, proper directory structure and permissions must be set up before launching the service.
- "sudo mkdir -p /var/log/opensearch /var/lib/opensearch"
- "sudo chown -R opensearch:opensearch /var/log/opensearch /var/lib/opensearch"
- "sudo chmod -R 755 /var/log/opensearch /var/lib/opensearch"
- Creates required directories for logs and data storage.
- Assigns ownership to the
opensearch
user to prevent permission errors.
- "sudo mkdir -p /etc/opensearch /var/log/opensearch /var/lib/opensearch"
- "sudo chown -R opensearch:opensearch /etc/opensearch /var/log/opensearch /var/lib/opensearch"
- "sudo chmod -R 750 /etc/opensearch"
- Ensures OpenSearch has the necessary configuration files in
/etc/opensearch/
. - Uses
chmod 750
to limit access, ensuring security-sensitive files are not world-readable.
Handling the Synonym List
One of the most difficult parts of the OpenSearch setup was ensuring that the custom synonym list was transferred and loaded correctly across all nodes.
- "sudo mkdir /etc/opensearch/synonyms/"
- Creates a dedicated directory for synonym management inside OpenSearch.
- "scp /home/bvest/dev/CodexMCP/synonyms/synonyms.txt codexmcp@{{NODE_IP}}:/tmp/synonyms.txt"
- Transfers the synonym list from the automation source directory (
CodexMCP/synonyms/
) to the target OpenSearch node.
- "sudo mv /tmp/synonyms.txt /etc/opensearch/synonyms/synonyms.txt"
- "sudo chown opensearch:opensearch /etc/opensearch/synonyms/synonyms.txt"
- "sudo chmod 644 /etc/opensearch/synonyms/synonyms.txt"
- Moves the file to its permanent location.
- Ensures OpenSearch has ownership and read access.
The synonym list is a key feature for optimizing search relevance and query interpretation, making it a necessary part of OpenSearch’s configuration.
Configuring OpenSearch
- "sudo sh -c 'cat <<EOF > /etc/opensearch/opensearch.yml\n{{OPENSEARCH_CONFIG}}\nEOF'"
- Replaces the default
opensearch.yml
with a custom configuration defined in{{OPENSEARCH_CONFIG}}
. - This ensures all nodes have consistent settings, including cluster formation, security, and indexing policies.
Starting OpenSearch
- "sudo su -c 'systemctl daemon-reload && systemctl start opensearch'"
- Reloads systemd to recognize OpenSearch as a service.
- Starts OpenSearch for the first time.
Summary
Deploying OpenSearch required a fully automated setup with multiple configuration steps. The most challenging parts included:
- Ensuring proper repository setup for installation.
- Managing directory permissions to prevent startup failures.
- Transferring and configuring the synonym list across multiple nodes.
Now that OpenSearch is fully automated, the next step is integrating it with the rest of CodexMCP, ensuring logs and indexed data are processed efficiently.