Diagnosing Linux network latency with mtr and iperf
Network latency can resemble bandwidth loss. An application may respond slowly because round-trip time is high, packets are dropped, jitter disrupts real-time traffic, or a saturated link lacks capacity. mtr exposes the route and per-hop timing behaviour, while iperf3 measures throughput and tests performance under load. Used together, they separate routing delay from congestion, packet loss, or a destination bottleneck.
Install both tools on RHEL, CentOS Stream, or Fedora with sudo dnf install -y mtr iperf3; use yum on older CentOS releases. Test only hosts you own or are authorised to assess. Keep the same client, endpoint, and test window for comparisons, and record the expected application response before changing the system.
Establish a baseline and map the path
First establish whether the slowdown affects every destination or only one service. Run ping -c 20 destination, inspect ip -s link, and use ss -ti to look for retransmissions, congestion signals, or a saturated socket. A high delay to a nearby router suggests a local problem; consistent delay to a remote host points toward the route beyond the gateway.
Australia’s geography can make route variation significant: traffic between Melbourne, Sydney, Brisbane, and Perth may cross several carrier networks. The local market includes NBN Co’s wholesale access, retailers such as Telstra, Optus, and Aussie Broadband, and regional fixed-wireless providers. CGNAT on some Australian fixed-wireless and mobile services can also prevent inbound iperf3 tests, so use a reachable VPS or an authorised public endpoint.
Choose an endpoint that represents the affected path. Save DNS, interface, routing, and system-load details, then repeat the baseline during a quiet period and at the time users report trouble. This turns an intermittent complaint into comparable evidence.
Interpret mtr without blaming every red hop
mtr combines traceroute with repeated probes. On RHEL-family systems, produce a ten-cycle report with:
mtr --report --report-cycles 10 203.0.113.20
The Loss%, Avg, Best, Wrst, and StDev columns show whether packets are missing and how much response time varies. A large standard deviation indicates jitter. Compare the result with a 20-packet ping of the final destination rather than judging the path from one trace.
Intermediate routers may rate-limit ICMP, so reported loss at a single middle hop can be an artefact. Persistent loss at the destination, matching ping loss, and growing RTT are stronger evidence of a real fault. Run a TCP trace when a firewall filters ICMP:
mtr -T -P 443 -r -c 20 203.0.113.20
Use --no-dns if reverse lookups slow the display, and avoid launching many probes against a fragile link.
Measure bandwidth and capacity with iperf3
On a controlled server, run iperf3 -s; from the client, run iperf3 -c 203.0.113.20. The report shows sender and receiver bitrate, RTT, and TCP retransmits. A large gap between offered and delivered traffic identifies a bottleneck more reliably than a single download speed test.
Reverse the flow and compare a single stream with parallel flows:
iperf3 -c 203.0.113.20 -R
iperf3 -c 203.0.113.20 -P 4
If one flow is slow but four reach the expected rate, the path may support more throughput than the application obtains. A UDP run at a known rate, iperf3 -c 203.0.113.20 -u -b 20M, exposes loss and jitter under load; inspect the receiver’s loss percentage.
Run the iperf3 server on port 5201 only on controlled systems. Open firewall access temporarily from the client’s known address and close it afterwards. Use the external testing reference to compare conditions, but treat measurements from your own endpoint as the baseline.
Use a repeatable troubleshooting checklist
Run the same sequence whenever conditions change:
- Compare the same destination with
mtr, endpoint ping, and oneiperf3flow; save each report. - Confirm whether loss reaches the destination instead of treating an isolated ICMP-limited hop as a failure.
- Test forward and reverse traffic, then compare one TCP flow with four parallel flows.
- Repeat at Australian peak periods, including weekday VPN traffic and evening home streaming.
- Check queues, retransmissions, interface errors, DNS, and firewall rules before altering routes.
- Change one variable at a time, record the result, and restore unrelated settings.
If only the application is slow, measure DNS, TCP, TLS, and time to first byte with curl -w. For Nginx, compare access-log timing with upstream response timing. The web server configuration guide covers proxy and timeout settings that can create apparent network delay. Check firewall rules with firewall-cmd --list-all or nft list ruleset, and expose iperf3 only to the temporary test source.
Combine the evidence into a diagnosis
High RTT with steady throughput usually indicates path distance or routing; increasing delay under load suggests congestion. Loss and retransmissions across several cycles indicate a less reliable path, while excellent network tests paired with slow application timing move attention to DNS, TLS, middleware, or the application. Re-run the decisive test after each change to prove the diagnosis.
Australian carriers operate under the Telecommunications Act 1997 and oversight from the Australian Communications and Media Authority, so measurements should remain on authorised infrastructure and respect applicable privacy rules. If logs contain personal information, your organisation’s obligations under the Privacy Act 1988 may also apply.
Remember that mtr locates delay and loss patterns along a route, while iperf3 verifies the usable capacity of that path; together they replace guesswork with a defensible diagnosis.