Automating OpenSearch Dashboards & Grafana Deployment with Ansible

Overview

Setting up monitoring and search interfaces manually is inefficient when working with constantly evolving infrastructure. To automate this process, we leveraged Ansible to deploy OpenSearch Dashboards and Grafana on interfaces-core-1. This guide details the exact steps taken, the reasoning behind them, the problems encountered, and the solutions implemented.

Though these are not core components of the finished system, they are still systems that we will use for design, testing, and troubleshooting throughout this project's development.

Why Automate This Deployment?

Manually installing, configuring, and maintaining OpenSearch Dashboards and Grafana across multiple servers is time-consuming and error-prone. Ansible allows us to:

• Ensure consistency across deployments.
• Dynamically configure OpenSearch connections using inventory variables.
• Automate repository setup for OpenSearch and Grafana.
• Enable services on boot to avoid manual intervention after reboots.

This approach is repeatable, scalable, and ensures that infrastructure remains as code rather than a manual process.

The Ansible Playbook: Installing OpenSearch Dashboards & Grafana

The playbook installs and configures OpenSearch Dashboards and Grafana, ensuring both are properly set up and running.

Prerequisites

• A working Ansible control node.
• The target server interfaces-core-1 defined in the Ansible inventory.
• OpenSearch nodes defined in the opensearch_cluster group.

The Playbook

- name: Install OpenSearch Dashboards & Grafana on interfaces-core-1
  hosts: interfaces-core-1
  become: true
  vars:
    osd_version: "2.18.0"
    osd_arch: "x64"
  tasks:

    - name: Ensure required dependencies are installed
      apt:
        name:
          - lsb-release
          - ca-certificates
          - curl
          - gnupg2
        state: present
        update_cache: yes

    - name: Download OpenSearch Dashboards Debian package
      get_url:
        url: "https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/{{ osd_version }}/opensearch-dashboards-{{ osd_version }}-linux-{{ osd_arch }}.deb"
        dest: "/tmp/opensearch-dashboards-{{ osd_version }}-linux-{{ osd_arch }}.deb"
        mode: '0644'

    - name: Install OpenSearch Dashboards
      apt:
        deb: "/tmp/opensearch-dashboards-{{ osd_version }}-linux-{{ osd_arch }}.deb"

    - name: Generate OpenSearch Hosts Array from Inventory
      set_fact:
        opensearch_hosts: "[{% for host in groups['opensearch_cluster'] %}\"http://{{ hostvars[host]['ansible_host'] }}:9200\"{% if not loop.last %}, {% endif %}{% endfor %}]"

    - name: Configure OpenSearch Dashboards
      template:
        src: opensearch_dashboards.yml.j2
        dest: /etc/opensearch-dashboards/opensearch_dashboards.yml
        owner: opensearch-dashboards
        group: opensearch-dashboards
        mode: '0644'
      notify: Restart OpenSearch Dashboards

    - name: Enable and start OpenSearch Dashboards service
      systemd:
        name: opensearch-dashboards
        enabled: yes
        state: started

  # ---------------- Grafana Installation ----------------

    - name: Create keyring directory for Grafana
      file:
        path: /etc/apt/keyrings/
        state: directory
        mode: '0755'

    - name: Import Grafana GPG key
      shell: |
        wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | tee /etc/apt/keyrings/grafana.gpg > /dev/null
      args:
        creates: /etc/apt/keyrings/grafana.gpg

    - name: Add Grafana Stable Repository
      shell: |
        echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list
      args:
        creates: /etc/apt/sources.list.d/grafana.list

    - name: Update APT cache after adding Grafana repo
      apt:
        update_cache: yes

    - name: Install Grafana OSS
      apt:
        name: grafana
        state: present

    - name: Enable and start Grafana service
      systemd:
        name: grafana-server
        enabled: yes
        state: started

  handlers:
    - name: Restart OpenSearch Dashboards
      systemd:
        name: opensearch-dashboards
        state: restarted

Challenges & Fixes

During deployment, we encountered a few issues that required debugging and fixes:

1. OpenSearch Dashboards failed to start due to an invalid authentication type (disabled).
   • Fix: Instead of setting opensearch_security.auth.type: "disabled", we used opensearch_security.enabled: false to properly disable security.
2. OpenSearch Dashboards didn't restart automatically after configuration changes.
   • Fix: We used Ansible's notify mechanism to restart the service only when the configuration file changed.
3. Grafana required manual repository configuration before installation.
   • Fix: The playbook was updated to import the GPG key, add the repository, and ensure the cache was updated before installation.

These fixes ensure the deployment is stable and fully automated.

OpenSearch Dashboards Configuration File

The playbook uses a Jinja2 template to populate with OpenSearch nodes from the inventory dynamically.

Template File: opensearch_dashboards.yml.j2

server.host: "0.0.0.0"

opensearch.hosts: {{ opensearch_hosts }}

opensearch_security.enabled: false

Important: These settings disable authentication in OpenSearch Dashboards. This is acceptable for testing environments only. Proper security configurations must be implemented before moving to production.

Next Steps

1. Secure OpenSearch Dashboards & Grafana
   • Right now, authentication is disabled. We need to enable OpenSearch Security and configure proper access controls.
2. Set Up Data Sources in Grafana
   • Grafana needs to be configured to pull data from OpenSearch.
3. Automate Further
   • Additional dashboards and data sources should be provisioned automatically.

This setup provides a solid foundation, but security and data visualization need further improvements. More work ahead, but we’re moving in the right direction.

This guide serves as a reference for future deployments, ensuring that we can replicate and refine the setup as the project evolves.