Stream editing with sed for shell automation on Linux

Sed quietly does heavy lifting in countless production shell scripts. Short for stream editor, it reads text line by line, applies transformations, and writes the result out — no prompts, no GUI. For admins juggling RHEL or CentOS boxes, it handles batch log cleanup, fleet-wide config tweaks, and data preprocessing.

In Australian operations rooms from Parramatta to Brisbane, scripts often handle timezone-aware logs, mining telemetry exports, or ATO-style batch submissions arriving in awkward CSV shapes. Sed slots into any pipeline and has shipped with Linux since well before most of us started, so the commands below work on a small VPS or a fleet in ap-southeast-2.

This walkthrough focuses on practical sed usage inside scripts: substitution patterns, addressing, in-place edits, deletes and inserts, plus the gotchas that catch people new to scripting. Examples target Fedora, RHEL and CentOS environments, and every snippet drops into a shell script or a cron one-liner.

Substitution basics and delimiter choices

The workhorse command is s/pattern/replacement/. By default sed operates on every line and prints the result, so sed 's/foo/bar/' input.txt > output.txt swaps the first match per line. Add the g flag to replace every occurrence, and use ^ or $ to anchor to start or end.

When your pattern contains slashes — paths like /etc/nginx/conf.d or URLs — switch the delimiter: sed 's#/var/log/nginx#/var/log/nginx-archive#' nginx.conf. Picking a delimiter that does not appear in the data avoids the noisy escape characters that turn scripts into regex hieroglyphics.

In-place editing versus piping

Two approaches matter for scripts: piping data through sed, or editing files directly with the -i flag. Piping is the safe default because the original file stays untouched, which is what you want during a dry run: cat hosts.conf | sed 's/old_host/new_host/' > hosts.conf.new.

For unattended scripts where you want to mutate the file, sed -i 's/old_host/new_host/' /etc/myapp/hosts.conf writes the change back. Passing -i.bak keeps a backup — a habit worth keeping when running over config files on a production box at 3am AEST.

Addressing specific lines and ranges

Sed addresses let you target only the lines you care about. A single number picks one line, a range like 10,20 picks a block, and a pattern match like /^#/ selects comment lines. Combining these keeps changes surgical: sed -i '/^#/!s/Port 22/Port 5022/' /etc/ssh/sshd_config skips commented lines and changes the active port.

This is useful when you maintain nginx configs that ship with a long block of commented defaults. Rewrite only the uncommented directives while leaving documentation intact, and pair the change with the nginx web server guides when documenting it for the team.

Deleting, inserting and appending lines

The d command deletes matched lines — the usual tool for stripping blank lines (sed '/^$/d' file) or dropping comments (sed '/^#/d' file). The i and a commands insert text before or after a matched line, useful for stamping headers onto rotated logs or appending footers to generated reports.

Stamping the current AEST timestamp into a daily export is one line: sed "1i\\# Generated $(date)" report.csv. Inside a script, wrap that in double quotes carefully so the date expands at runtime rather than getting baked in when the script is written.

Running sed inside larger shell scripts

When sed becomes one stage in a longer pipeline, treat it like any other command: capture exit status, redirect errors, and quote variables. A nightly script might pull logs, rewrite paths, deduplicate, and archive: journalctl -u nginx --since "yesterday" | sed 's/oldhost/newbackend/' | sort -u | gzip > /backup/nginx-$(date +%F).gz.

Function wrappers help readability. A conf_replace() function taking a file, pattern and replacement keeps calling code clean. Publishing shared helpers in a shared utility repo lets colleagues reuse them without copy-pasting.

Combining sed with awk and grep

Sed shines for line-by-line rewrites, but it is not the right tool for column extraction — that is where awk takes over. Use grep as a coarse upstream filter to cut input volume: grep -E 'ERROR|WARN' /var/log/app.log | sed 's/^.*: //' | awk '{print $1}' | sort | uniq -c.

The order matters. Filtering with grep reduces the data sed has to scan, which on a busy server crunching gigabytes of mining telemetry files can shave real minutes off a job that runs before the Sydney office opens.

Portability across RHEL, CentOS and Fedora

GNU sed on RHEL-family distributions supports extensions such as -E for extended regex, -i for in-place editing, and \b for word boundaries. Older POSIX sed on minimal containers may not have those, so for maximum portability stick to basic regex and avoid GNU-only flags.

When scripts run on legacy CentOS 7 and modern Fedora 40 setups, walking through a Fedora 40 LAMP guide confirms identical sed behaviour. Prefer single-quoted sed expressions over double-quoted ones, and avoid process substitution. Test in a stripped container before promoting to cron, saving you from a 6am call when the AEDT changeover hits.

Practical habits for sed-driven scripts

• Quote every sed expression and treat user input as untrusted data.
• Use -n with p to keep output volume predictable.
• Prefer extended regex (-E) when patterns get nested.
• Always test with a pipe first, then move to -i once the output looks right.
• Keep a backup suffix on -i for any change touching /etc or app data.
• Validate with set -euo pipefail so a failed sed stops the pipeline.
• Log what was changed so post-incident reviews on a busy Tuesday arvo are not guesswork.

Sed is one of the smallest, fastest and most dependable tools in any Linux admin's kit. A single line of sed can replace dozens of lines of Python for everyday jobs on RHEL, CentOS and Fedora. Treat it as a building block, lean on awk when columns matter, and always keep a backup before letting it loose on a production file.