Filtering systemd logs with journalctl
systemd-journald collects structured, timestamped records from services, the kernel, user-space processes, and startup tasks. On RHEL, CentOS, Fedora, and related systems, journalctl is the supported interface for querying that journal without manually opening files beneath /var/log.
Across Australia’s cloud hosting market and internal estates, effective log filtering turns routine triage into a repeatable process. Narrowing by service, event time, priority, process, or boot session reveals relevant evidence while keeping unrelated entries out of the investigation.
How journal filters work
Filters operate on journal metadata as well as message text. A unit filter uses structured unit fields, while options such as --since and --until constrain timestamps. Combining filters creates a focused view, and output controls such as -o short-iso preserve useful context without the width of full journal entries.
Field filters are more reliable than message searches when metadata is available. For example, journalctl _PID=812 selects records indexed for process 812, whereas grep searches the rendered output. Use the first approach for identity, and reserve grep for distinctive error wording that has no indexed field.
Time, units and priorities
--since and --until accept fixed dates, relative expressions such as 30 min ago, and clock-based expressions. A bounded query such as journalctl --since "2026-10-18 09:00" --until "2026-10-18 10:00" -u nginx.service limits results to one service during a specific hour.
Australian operations make time boundaries especially important. Sydney and Melbourne generally switch between AEST (UTC+10) and AEDT (UTC+11), Brisbane remains on AEST, and Perth uses AWST (UTC+8); an unqualified date is normally interpreted in the machine’s local zone. Before ordering events across Australian hosts, verify chrony time synchronisation because clock skew can make correct entries appear out of sequence.
The -u option accepts one or more units, including dependencies when named explicitly. Add -p err when you want errors and more severe events rather than routine informational messages. Leaving priority off is often better for authentication or application incidents, where normal records provide the sequence needed to explain a failure.
Fields, identifiers and boot sessions
Useful indexed fields include _SYSTEMD_UNIT, _PID, _UID, _COMM, _EXE, _HOSTNAME, and SYSLOG_IDENTIFIER. Combine them cautiously: journalctl _UID=0 _COMM=sudo --since today is useful for sudo activity, but _PID alone can be ambiguous if a process identifier is reused. Quote values containing spaces.
Boot filters separate incidents across restarts. journalctl -b reads the current boot, while journalctl -b -1 reads the previous one; -k restricts results to kernel messages. Combine these as journalctl -k -b -1 when investigating a panic, filesystem error, or memory-pressure event that preceded a reboot.
Tracing common incidents
For an SSH authentication failure in Melbourne, begin with journalctl -u sshd.service --since "09:00" --until "10:00". If several units interact, request them together with repeated -u options, then pipe the result through grep -E "failed|invalid|timeout" only when indexed filters cannot express the wording you need.
For web-service failures, compare nginx.service with its resolver, network, or mount dependencies. A useful pattern is journalctl -u nginx.service -u systemd-resolved.service --since "30 min ago" | grep -Ei "timeout|refused|unreachable". Preserve the unfiltered service output separately when possible, because a keyword match may omit the event that explains the matching line.
Persistent storage and retention
By default, some RHEL-family configurations keep journals in volatile storage such as /run/log/journal, which disappears at shutdown. Set Storage=persistent in /etc/systemd/journald.conf when logs must survive reboots, then restart systemd-journald; check /var/log/journal and its permissions before relying on retention for incident response.
Control growth with settings such as SystemMaxUse=2G and MaxRetentionSec=1month, then inspect consumption with journalctl --disk-usage. Older files can be removed with journalctl --vacuum-size=2G or journalctl --vacuum-time=30d. Treat retention as an operational balance: enough history for investigations, without consuming disk needed by databases or monitoring agents.
Comparing filter approaches
No single filter answers every incident. Select the narrowest reliable criterion, then add another only when the result needs more context.
| Approach | Example | Best suited to | Limitation |
|---|---|---|---|
| Unit | -u nginx.service |
One service’s lifecycle | Misses dependency messages |
| Priority | -p err |
Failures and severe events | Removes normal context |
| Field | _PID=812 |
Process or user identity | Identifiers may be reused |
| Boot | -b -1 |
Incidents before a restart | Needs a surviving journal |
| Kernel | -k |
Storage, driver, memory events | Excludes user-space services |
| Follow | -f |
Live troubleshooting | Can omit earlier evidence |
Combine methods instead of forcing one: a previous-boot unit query plus a time window and a field match is usually stronger than a broad text search. For automation, add --no-pager; for evidence or downstream parsing, -o json retains fields more consistently than human-formatted output.
A dependable response workflow
A practical response sequence keeps the first query simple, records what the current boot contains, and expands only when the evidence requires it.
- Record the incident’s local start and end times.
- Start with the affected unit or boot session.
- Add priority only when routine messages are irrelevant.
- Correlate process, user, and host fields on shared systems.
- Check clock health before comparing event order.
- Confirm retention before promising access to old logs.
Capture the relevant window before restarting a service or host, especially when storage is volatile. For example, save the output of journalctl -u nginx.service -b -1 -o short-iso before reboot erases the previous session. The practical takeaway is to preserve a verified time window, narrow it with structured identity, and export the evidence while the boot record remains available.