CodexMCP Automation: rsyslog setup

Overview

This section defines how the rsyslog cluster nodes are installed and configured. The goal is to create a centralized logging server that collects logs from all infrastructure components. This is still an alpha-stage prototype, meaning that this is what works right now, but it will evolve as the project progresses.

The steps below handle the installation of rsyslog, enable remote log reception, configure log storage, and ensure proper permissions and service management.

Update and Install Rsyslog

  - "sudo apt-get update && sudo apt-get install -y rsyslog"

This ensures that rsyslog is installed and up to date on the system. The update command fetches the latest package lists from the distribution's repositories, and the installation command installs rsyslog if it is not already present.

Enable UDP and TCP Reception

  - "sudo sed -i 's/#module(load=\"imudp\")/module(load=\"imudp\")/' /etc/rsyslog.conf"
  - "sudo sed -i 's/#input(type=\"imudp\" port=\"514\")/input(type=\"imudp\" port=\"514\")/' /etc/rsyslog.conf"
  - "sudo sed -i 's/#module(load=\"imtcp\")/module(load=\"imtcp\")/' /etc/rsyslog.conf"
  - "sudo sed -i 's/#input(type=\"imtcp\" port=\"514\")/input(type=\"imtcp\" port=\"514\")/' /etc/rsyslog.conf"

These commands enable rsyslog to accept logs over both UDP and TCP on port 514. By default, rsyslog has these options commented out in its configuration file. The sed command removes the comment markers (#), effectively enabling the relevant settings.

UDP is a connectionless protocol that is faster but less reliable, while TCP ensures message delivery at the cost of some overhead. Supporting both allows flexibility in how different devices send their logs.

Ensure the Remote Log Storage Directory Exists

  - "sudo mkdir -p /var/log/remote"
  - "sudo chown syslog:adm /var/log/remote"
  - "sudo chmod 750 /var/log/remote"

These commands create the directory where logs from remote systems will be stored.

• The mkdir -p command ensures the /var/log/remote directory exists, creating it if necessary.
• The chown command sets the owner of the directory to the syslog user and the adm group, ensuring rsyslog has the correct permissions to write logs.
• The chmod 750 command sets permissions so that only the owner (syslog) and group members (adm) can access the logs, while others are denied access.

Configure Remote Logging Directory

  - "echo 'template(name=\"RemoteLogs\" type=\"string\" string=\"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log\")' | sudo tee -a /etc/rsyslog.conf > /dev/null"
  - "echo '*.* ?RemoteLogs' | sudo tee -a /etc/rsyslog.conf > /dev/null"
  - "echo '& stop' | sudo tee -a /etc/rsyslog.conf > /dev/null"

These commands define how remote logs are stored on the rsyslog server.

• The first command creates a template named RemoteLogs that determines the format for storing logs. Logs are saved in /var/log/remote/, organized by hostname (%HOSTNAME%) and program name (%PROGRAMNAME%). This keeps logs structured and easy to navigate.
• The second command tells rsyslog to use this template for all incoming log messages. The *.* syntax ensures that logs from all sources and all severity levels are stored using this format.
• The third command stops further processing of logs once they are written to the appropriate file, preventing duplicate entries or unwanted processing loops.

Restart the Rsyslog Service

  - "sudo systemctl restart rsyslog"

This applies all the configuration changes by restarting the rsyslog service. Restarting ensures that any modifications to the configuration file take effect immediately.

Summary

This section sets up the rsyslog cluster nodes to accept remote logs, store them in an organized manner, and prevent unnecessary duplication or processing loops. By automating these steps, every logging server in the cluster can be provisioned quickly with a standardized configuration.

The next section will document how other core services integrate with this logging system, ensuring that logs from all infrastructure components are properly collected and indexed.