Setting up a centralized Syslog server with Logstash

Centralized logging gives Linux administrators one searchable record of host events, network warnings, and security activity. A practical design uses rsyslog to ship messages to Logstash, Elasticsearch for indexed storage, and Kibana for dashboards, with Beats covering application files that never entered Syslog.

This guide uses RHEL, CentOS, Fedora, or a compatible Linux host. Treat collection as a security system: use UTC timestamps, restrict network access, retain only justified data, and test recovery instead of assuming successful delivery means complete coverage.

Designing the logging path

Map the path before installing packages: Linux hosts and appliances send Syslog to a hardened collector; Logstash parses and enriches events; Elasticsearch stores indexed documents; Kibana provides role-controlled views. Include network devices, authentication services, and auditd or journald records when they form part of the incident picture.

Australian teams often span Sydney, Melbourne, Brisbane, and Perth, while hybrid work makes remote role-based access routine; normalised UTC timestamps prevent daylight-saving ambiguity. The design also suits the country’s banking, mining, and gaming markets. A casino and pokies operator may need a defensible trail across websites, point-of-sale systems, and venue infrastructure.

Preparing the collector host

Use a dedicated server or VM with stable DNS, sufficient disk, and monitored RAM. Give Logstash and Elasticsearch separate non-root accounts, protect private keys, and keep the collector off developer desktops. In Australia, a managed service may suit smaller organisations, but self-hosting gives local operators direct control over retention and outages.

Open only required firewall ports to approved source networks. For a firewalld host, add 6514/TCP for encrypted Rsyslog traffic and 5601/TCP only if Kibana must be reached directly; otherwise keep Kibana behind an authenticated reverse proxy. Check SELinux port labels before changing them rather than disabling the service.

Installing the Elastic Stack

Add Elastic’s repository for the distribution and install matching component versions, then start them with systemd. A typical RHEL-family installation is sudo dnf install elasticsearch logstash kibana, followed by sudo systemctl enable --now elasticsearch logstash kibana. Confirm Elasticsearch responds locally and let the cluster health settle before shipping events.

Set an initial heap size no larger than half the available RAM, keep JVM memory equal, and leave free disk for indexing. Version-control configuration, but keep certificates, passwords, and API keys in restricted files or secret injection. Back up Elasticsearch snapshots and document a rebuild procedure rather than treating backups as optional.

Forwarding events with rsyslog

On each source, create /etc/rsyslog.d/10-forward.conf and enable the journal, Unix socket, and relay modules. A basic TLS target looks like this:

module(load="imjournal")
module(load="imuxsock")
module(load="omrelay")

action(
  type="omrelay"
  target="logstash.example.net"
  port="6514"
  protocol="tcp"
  template="RSYSLOG_TraditionalForwardFormat"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverAuth="x509/name=logstash.example.net"
  StreamDriverPermittedPeers="logstash.example.net"
)

Choose the transport deliberately:

Method	Main advantage	Main limitation	Suitable for
TCP/TLS on 6514	Encryption, per-sender ordering and acknowledgements	Requires certificates and careful firewall rules	Linux hosts forwarding to Logstash
UDP on 514	Low overhead and broad appliance support	No delivery guarantee; packets may be lost	Network devices with local buffering
Filebeat or another Beat	Structured JSON and a local retry queue	Adds another agent to maintain	Applications and containers

Prefer TCP/TLS for Linux fleets because it provides ordering, delivery checks, and encryption; UDP is simpler for constrained appliances but can lose events. Validate with sudo rsyslogd -N1 before restarting the daemon, and verify that the certificate name matches the collector’s DNS name.

Parsing and enriching events

Configure Logstash as a pipeline rather than a filter script hidden inside shell commands. The input receives TLS-protected Syslog; filters parse structured fields; the output writes to Elasticsearch and, briefly, standard output for testing.

input {
  tcp {
    port => 6514
    ssl => true
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.key"
    ssl_client_authentication => "required"
    ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
  }
}
filter {
  grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{GREEDYDATA:message}" } }
  date { match => ["syslog_timestamp", "MMM  d HH:mm:ss", "MMM d HH:mm:ss"] }
}
output {
  elasticsearch {
    hosts => ["https://127.0.0.1:9200"]
    api_key => "${LOGSTASH_ES_API_KEY}"
    ssl_certificate_authorities => ["/etc/pki/tls/certs/ca.crt"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

Use tags and conditional rules to classify authentication failures, firewall denials, and hardware faults without discarding unmatched messages. Test regular expressions before deployment; admins who prefer Emacs can use the Emacs pattern matching guide to refine expressions before translating them into Grok patterns. Send parse failures to a monitored queue rather than hiding them.

Securing and validating the pipeline

Use mutual TLS where practical, give each source a distinct identity, and permit traffic only from managed networks. Protect Elasticsearch and Kibana with authentication, least-privilege roles, and encryption in transit. Keep the API key for Logstash separate from human administrator credentials, and rotate it on a defined schedule.

Validate end to end with logger, journalctl, and a Kibana data view. Compare source counts with indexed counts, inspect malformed events, and confirm that a blocked host cannot bypass the firewall. Run the test after certificate, disk, or network changes, not only during installation.

Logs can contain IP addresses, usernames, and customer identifiers that are personal information under the Privacy Act 1988 and the Australian Privacy Principles. State and territory privacy laws may also apply. Minimise sensitive fields, restrict exports, and set retention against documented operational, contractual, and legal needs; the federal law does not create one universal log-retention period.

Operating the logging service

Operate the service as a long-running data pipeline. Watch Elasticsearch disk watermarks, JVM pressure, Logstash queue growth, and failed TLS handshakes; route system alerts to a separate channel. Use index lifecycle management or data streams to roll hot logs to cheaper storage and delete expired data without manual cleanup.

Measure event loss during a controlled relay outage, then document recovery, certificate renewal, and capacity thresholds. Keep collector configuration under change control and rehearse restoration from an Elasticsearch snapshot on a separate host.

The practical takeaway is to begin with one hardened collector, a documented flow, and an end-to-end delivery test; add sources only after alerts, retention, and recovery have been proven.