Tracing Linux behaviour with strace

strace is a Linux diagnostic utility that records the system calls made by a process. It exposes the boundary between an application and the kernel, showing which files are opened, which sockets are created, and why an operation fails. That visibility turns vague symptoms such as a slow start or denied connection into evidence you can inspect. Learn more about Hardening Linux With Sysctl Kernel Parameters.

On RHEL, Fedora and CentOS Stream systems, begin with sudo dnf install strace, then trace a harmless test command before touching a service. This workflow suits Australian enterprise environments, where RHEL estates, Fedora engineering workstations and container platforms often coexist.

Seeing what a process actually does

Run strace with the command you want to investigate: sudo strace -o trace.log /usr/bin/your-application. The log contains the arguments and results of each syscall, including execve, openat, read, connect and exit_group.

An openat call returning file descriptor 3 shows that a file was opened successfully. A connect returning 0 means the network connection was established, while -1 followed by an error such as ECONNREFUSED explains the failure. The read and write entries then reveal whether data actually moved.

To attach to an existing process, use sudo strace -f -p 1234. The -f option follows child processes and threads created with fork, vfork or clone. Functional checks with browser-driven test suites can confirm overall behaviour, while strace shows the kernel-level activity behind that result.

Filtering noise from the trace

An unrestricted trace can become enormous. Start with a focused expression such as strace -e trace=openat,connect,read,write, or use the broader strace -e trace=%file,%network,%process,%memory when investigating mixed application behaviour.

The result column carries the most important diagnostic evidence. ENOENT indicates a missing path, EACCES indicates a permissions problem, and ETIMEDOUT suggests that an operation exceeded its allowed time. Filtering with -e status=successful hides failures and is useful for reviewing normal flow, but it should not be the first filter used during fault diagnosis.

Timestamps help connect the trace with application logs. Use -tt for readable local times, -ttt for epoch timestamps and -T for syscall duration. Australian teams should record UTC alongside AEST or AEDT because daylight-saving changes can make local logs ambiguous, while the familiar day/month/year convention can complicate comparisons with ISO-formatted records.

Tracing services and containers

For a service, first reproduce the fault in staging or on a maintenance host. Run the service beneath strace with sudo strace -f -o /var/log/service.trace /usr/bin/service --config /etc/service.conf. Attaching to a production PID is possible, but the trace adds overhead and may expose sensitive material.

Containers require an additional namespace decision. After following Docker on Fedora, you can trace a container’s main process from the host with sudo strace -f -p $(docker inspect -f '{{.State.Pid}}' app). Alternatively, install strace inside the container and use docker exec; host and container views may expose different paths and capabilities.

Australian-hosted services often span Sydney and Melbourne regions, use .au domains and connect through global content-delivery networks. A trace can reveal the actual address and network family reached, rather than relying on assumptions based on the hostname or its apparent geographic location.

Keeping traces safe and proportionate

System-call tracing increases CPU use, latency and I/O, particularly when every read and write is recorded. Use -c for a syscall summary, narrow the trace to relevant calls and avoid tracing an entire busy workload. Capturing output to a protected file with chmod 600 trace.log also limits accidental disclosure.

Linux may restrict process attachment through ptrace_scope. Inspect the current setting with sudo sysctl kernel.yama.ptrace_scope; an appropriate user relationship or elevated privilege is required when attachment is denied. If the evidence points to networking or kernel controls, review sysctl kernel hardening alongside the trace instead of changing settings blindly.

Trace files may contain credentials, request bodies, customer records and file contents. Australian workloads handling personal information should account for the Privacy Act and OAIC guidance, while security controls should align with ASD Essential Eight practices. Restrict trace filters, redact sensitive values and keep diagnostic logs out of shared ticketing systems.

Turning a trace into a fix

A reliable investigation reproduces one failure, captures the smallest useful trace and identifies the syscall where the expected result diverged. File errors usually point to paths, permissions or mount problems; network errors point to name resolution, routing or service availability; unusually long calls often indicate blocked storage or slow remote endpoints.

Investigation goal	Useful strace options	Evidence to inspect	Main trade-off
Find configuration errors	-e trace=%file,%process	openat, stat and ENOENT	Narrow filters can hide related activity
Diagnose network failures	-e trace=%network	socket, connect, DNS calls and errno values	Payload calls may expose sensitive data
Measure I/O delays	-tt -T -e trace=read,write,fsync	Elapsed time and repeated file descriptors	Tracing adds latency to each call
Check permission failures	-e trace=%file,%creds	EACCES, credential changes and denied paths	Credential details require careful handling
Summarise overall activity	-c	Call counts, errors and total time	No sequential call context is retained

The practical takeaway is a small, timestamped trace tied to one reproduced failure, with every fix checked against the same controlled workload and every captured payload protected.