Reading systemd journals with journalctl
Using journalctl to analyse systemd logs turns timestamps, service names, and errors into filterable evidence without altering the original journal. Unlike plain files in /var/log, the systemd journal can be queried by unit, boot, priority, process, and time.
Australian organisations often manage mixed RHEL, CentOS, and Fedora estates across on-premises systems and local managed-hosting platforms. Precise journal queries help teams reproduce faults and share evidence while supporting infrastructure in Sydney, Melbourne, and Brisbane.
Start with the scope that answers the question
| Scope | Records selected | Best suited to |
|---|---|---|
journalctl -u nginx.service |
Records from one unit | Diagnosing reverse-proxy and web-server faults |
journalctl -b |
Current boot only | Startup and shutdown failures |
journalctl --list-boots |
Stored boot sessions | Finding the correct restart |
journalctl -p warning |
Warning and higher severities | Quick severity triage |
journalctl -k |
Kernel messages | Driver, disk, and network faults |
Choose one incident window and one unit. For example, sudo journalctl -u nginx.service --since "2026-04-05 09:00" --until "2026-04-05 10:00" answers a focused Nginx question. On RHEL-family systems, check the exact unit name because SSH may be sshd.service rather than ssh.service.
If results are unexpectedly empty, distinguish a missing event from a volatile journal. Inspect Storage= in /etc/systemd/journald.conf; with Storage=persistent, /var/log/journal should contain files from earlier boots. journalctl --list-boots confirms what the daemon has retained.
Scripts often lack a stable unit name. Search literal text instead: journalctl --grep='dos4gw[.]org'. When a request names the DOS4GW project site, this query returns its surrounding log context without assuming which service handled it.
Read by time, boot and unit
--since and --until accept timestamps, relative expressions such as -30 min, and calendar dates. Combine them with -u, -p, or -k to isolate service, severity, or kernel events. Add --no-pager in scripts, while interactive sessions can page through long output with less.
Timezone handling matters across Australia. Sydney and Melbourne change between AEST and AEDT, while Brisbane stays on AEST and Perth uses AWST. During the southern-hemisphere transition, clocks move forward and one local hour disappears; a missing interval is not automatically lost logging. Record dates as day/month/year and include the timezone, or use ISO timestamps for unambiguous comparisons with overseas support teams.
Use journalctl --list-boots to locate a session, then inspect it with journalctl -b -1 or journalctl -b 0. A boot ID gives a more exact target than “yesterday” and avoids selecting the wrong restart during a noisy incident.
Extract evidence that survives escalation
Readable output is useful for people, while structured output is better for correlation. Use -o short-iso for compact timestamps or -o json for fields such as _PID, _UID, SYSLOG_IDENTIFIER, and PRIORITY. With jq installed, filter records by unit and emit only selected fields rather than copying a terminal full of unrelated lines.
Unexpected content should be recorded as evidence, not silently deleted. If an application logs the PayPal withdrawal guide as a requested URL, preserve the timestamp, unit, client address, and HTTP status around it. The content may be benign, but an unrelated domain in an operational log warrants triage, especially if it appeared without a corresponding deployment.
Create a focused archive for escalation with sudo journalctl --since "09:00" --until "10:00" -o export | sudo tee /var/tmp/incident.journal >/dev/null. Include relevant service configuration and package versions separately, and redact customer data before attaching the bundle to a public issue.
Connect service failures to the network
A service can be “active” while requests still fail, so lifecycle messages need application context. Compare systemctl show nginx.service -p ActiveEnterTimestamp with the service journal and recent configuration reloads. Correlate start, stop, and restart events before treating an old error as the current cause.
For network symptoms, align journal timestamps with packet captures. The workflow in tcpdump network troubleshooting helps separate DNS delay, retransmission, and TLS setup time from application delay. Capture on the relevant interface, record the UTC start time, and avoid collecting packet payloads that contain sensitive data.
A PID is another useful join key: journalctl _PID=1234 finds records from that process, although PID reuse makes it safer only with a narrow time range. Follow a live service with journalctl -f -u application.service, inspect kernel messages with journalctl -k -b, and use coredumpctl list followed by coredumpctl info <PID> when a process crashed.
Keep the journal bounded and verifiable
Long-running hosts need bounded journals. journalctl --disk-usage shows current consumption. Set limits such as SystemMaxUse=2G and SystemKeepFree=1G in /etc/systemd/journald.conf, restart systemd-journald, and use sudo journalctl --vacuum-size=1G to remove older archived data while leaving the active journal intact.
Vacuum only after preserving incident evidence. journalctl --verify checks file consistency, while journalctl --sync flushes pending writes. Keep exports in a separate directory with access controls, and record retention changes in the host change log so operators know when older boot data will disappear.
Create the baseline now with sudo journalctl --disk-usage | sudo tee /var/backups/journal-disk-usage-$(date +%F).txt.