Saturday Midday Update: Lots of Automation Work

Introduction

Today was a major milestone in the automation of CodexMCP’s infrastructure. We tackled automated OpenSearch deployments, fine-tuned Cloud-Init provisioning, and streamlined VM bootstrapping within Proxmox.

The result? A fully deployed multi-node OpenSearch cluster in under 5 minutes, all built dynamically from YAML configurations with no manual intervention. This is a blend of old-school UNIX thinking and modern infrastructure automation, ensuring full visibility into system operations while eliminating unnecessary abstraction.

Here’s a deep dive into everything we accomplished.

1. Automating OpenSearch Deployment in Proxmox

The first major challenge was getting OpenSearch installed across multiple Proxmox-deployed VMs, without relying on Ansible or any external tools. Instead, we leveraged YAML-based configurations and Go-based VM bootstrap automation.

1.1. VM Configuration in vmconfig.yaml

Instead of hardcoding anything into the Go code, we designed a clear and explicit YAML configuration that defines every step of the deployment.

Example vmconfig.yaml

proxmox_host: "https://yourproxmoxhost:port"
proxmox_node: "proxmoxnode"
api_token: "env"

ssh:
  username: "codexmcp"
  private_key: "/home/youruser/.ssh/id_rsa"

logging_server:
  name: "logs-core-1"
  ipaddress: "10.0.1.100"

global_software:
  - "sudo apt update && sudo apt install -y rsyslog"
  - "sudo systemctl enable rsyslog && sudo systemctl start rsyslog"
  - "echo '*.* @{{LOGGING_SERVER_IP}}:514;RSYSLOG_ForwardFormat' | sudo tee /etc/rsyslog.d/50-remote.conf > /dev/null"
  - "sudo systemctl restart rsyslog"

clusters:
  opensearch_cluster:
    multi_node: true
    config_template: |
      cluster.name: codexmcp-cluster
      node.name: "{{NODE_NAME}}"
      network.host: "0.0.0.0"
      {{DYNAMIC_DISCOVERY}}
      plugins.security.disabled: true

    software:
      - "sudo apt-get update && sudo apt-get install -y lsb-release ca-certificates curl gnupg2"
      - "curl -fsSL https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo gpg --dearmor -o /usr/share/keyrings/opensearch-keyring.gpg"
      - "echo 'deb [signed-by=/usr/share/keyrings/opensearch-keyring.gpg] https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main' | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list > /dev/null"
      - "sudo apt-get update"
      - "sudo apt-get install -y opensearch"
      - "sudo mkdir -p /etc/opensearch"
      - "sudo chown -R opensearch:opensearch /etc/opensearch /var/log/opensearch /var/lib/opensearch"
      - "sudo chmod -R 750 /etc/opensearch"
      - "sudo systemctl daemon-reload"
      - "sudo systemctl enable --now opensearch"

    vms:
      - vmid: 2019
        name: "os-core-1"
        ipaddress: "10.0.1.151"
        gateway: "10.0.1.1"
      - vmid: 2020
        name: "os-core-2"
        ipaddress: "10.0.1.152"
        gateway: "10.0.1.1"
      - vmid: 2021
        name: "os-core-3"
        ipaddress: "10.0.1.153"
        gateway: "10.0.1.1"

This structure ensures absolute clarity on what’s happening at every step.

2. Breaking Down the Automation Code

We split the Go automation logic into modular, single-responsibility files:

.
├── APIAttachCloudInit.go
├── APICloneVM.go
├── APIConfigureNetwork.go
├── APIConfigureVM.go
├── APIProxmoxRequest.go
├── APIRemoveSSHKey.go
├── APIResizeDisk.go
├── APIStartVM.go
├── APIVmExists.go
├── bootstrapvms.go
├── BSLoadInventory.go
├── BSRunRemoteCommand.go
├── BSWaitForApt.go
├── BSWaitForSSH.go
└── OSBuildConfig.go

Each file only imports what it needs, keeping things lightweight, easy to debug, and scalable.

3. Fixing the OpenSearch Installation Issues

OpenSearch had a painful installation process that required the following fixes:

3.1. Disabling Security During Bootstrap

OpenSearch won’t install cleanly if security is enabled, so we inject this config:

plugins.security.disabled: true

via:

- "sudo sh -c 'cat <<EOF > /etc/opensearch/opensearch.yml\n{{OPENSEARCH_CONFIG}}\nEOF'"

3.2. Fixing OpenSearch Permissions

Even after installation, OpenSearch wouldn’t start due to permission errors.
We solved this by adding:

- "sudo mkdir -p /etc/opensearch /var/log/opensearch /var/lib/opensearch"
- "sudo chown -R opensearch:opensearch /etc/opensearch /var/log/opensearch /var/lib/opensearch"
- "sudo chmod -R 750 /etc/opensearch"

3.3. Ensuring OpenSearch Starts Correctly

We had to run:

- "sudo su -c 'systemctl daemon-reload && systemctl start opensearch'"

because regular sudo wasn’t enough in some cases.

4. Proxmox Cloud-Init Image Creation

We ran into issues with Ubuntu Server freezing during install because 1GB RAM was too low.
We fixed this by installing Ubuntu with 2GB RAM, then shrinking it for Cloud-Init.

Steps for a Clean Cloud-Init Template

Convert to a Proxmox Template

qm template 9000

Install Ubuntu (Minimal) and Cloud-Init

apt update && apt install -y cloud-init
cloud-init clean
shutdown -h now

Create a new VM

qm create 9000 --name ubuntu-template --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:10
qm set 9000 --cdrom local:iso/ubuntu-22.04-live-server-amd64.iso
qm set 9000 --boot order=ide2

Now, we can clone any number of VMs instantly from this template.

5. Final Deployment: Full Cluster Automation

Once the template and bootstrapper were ready, we deployed a full OpenSearch cluster with a single command:

go run main.go --bootstrapvms --cluster opensearch_cluster

After 5 minutes, we had a fully working OpenSearch cluster, all configured dynamically from YAML.

Final test:

curl 10.0.1.151:9200

Result:

{
  "name" : "os-core-1",
  "cluster_name" : "codexmcp-cluster",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.19.0"
  }
}

Conclusion

This approach eliminates unnecessary abstraction while ensuring fully automated deployments.
It’s a fusion of old-school sysadmin knowledge and modern DevOps principles—fully automated yet fully transparent.

-Remember This is all Alpha at this point
--Bryan