Rescue Mode on RHEL: Fixing Boot Failures When Systems Refuse to Start
When a Red Hat Enterprise Linux server refuses to come back up after a reboot, the admin on call usually has two choices. They can panic, or boot into the built-in rescue environment and work through the damage in a controlled way. Rescue Mode on RHEL is a stripped-down systemd target that mounts the local root filesystem under /mnt/sysimage and drops you into a shell where you can fix fstab entries, rebuild an initramfs, or repair a broken GRUB installation. It is the same workflow taught in RHEL courses at RMIT and the University of Sydney, and what senior engineers in Brisbane and Melbourne reach for before considering a reinstall.
Across Australian IT shops, the late arvo is often reserved for change windows, and that is when a botched dnf update is most likely to take a server down. Whether the hardware lives in a hyperscale facility in Sydney or a small colocation rack in Adelaide, knowing how to drop into rescue mode means you do not need to drag a keyboard and monitor down the hallway. The approach also lines up with practices around the Australian Cyber Security Centre's Essential Eight, where logged interventions and reproducible repair steps are exactly what auditors want to see.
This walkthrough covers the full sequence from entering the GRUB menu to reinstalling the bootloader and bringing the server back online. It also clarifies the differences between rescue, emergency, and single-user modes, because mixing them up is a common mistake when you are staring at a blank console at half past midnight AEDT.
What Rescue Mode Actually Does
The rescue target sits inside systemd's collection of boot targets, alongside multi-user.target and emergency.target. When you boot into rescue.target, systemd starts a minimal set of services, attempts to mount anything in /etc/fstab that is not flagged nofail, and hands you a root shell with the local disk available for editing. It does not provide networking by default, so anything you do has to happen on the console or via an out-of-band management card. For teams running bare metal in Perth data centres, that local-only behaviour is a feature: you cannot accidentally trigger remote replication while fixing a filesystem.
Reaching the GRUB Menu on RHEL
To enter rescue mode, interrupt the normal boot sequence. Power the machine on and hold Shift (BIOS) or tap Escape (UEFI) until the GRUB menu appears. Highlight the kernel you want to boot, press e to edit, and find the line beginning with linux or linux16. Append systemd.unit=rescue.target to the end of that line, then press Ctrl+X to boot. The system will run through POST and dracut as usual, then hand you a root shell. If the GRUB menu itself is broken, boot from the RHEL installation ISO and choose Troubleshooting → Rescue a Red Hat Enterprise Linux system, which gives you the same shell but with the disk mounted under /mnt/sysimage. Most operators bookmark these steps next to general Linuxtpoint runbooks for offline repairs.
Rescue vs Emergency vs Single-User Targets
It is worth pausing to compare these three targets, because they look similar on the surface but behave quite differently. Picking the right one early saves time during a midnight callout from a Canberra-based ops team.
| Target | Filesystems Mounted | Networking | Use Case |
|---|---|---|---|
| rescue.target | Yes, except nofail failures |
Disabled by default | Repair fstab, reinstall GRUB, run dnf rollback |
| emergency.target | No | Disabled | Root filesystem is broken or will not mount at all |
| single-user mode | Yes | Disabled | Password reset, minor config edits when root is intact |
If fstab references a mount point that no longer exists, rescue mode will refuse to drop you into the shell. Emergency mode skips mount attempts entirely and gives you a raw shell, which is exactly what you need when the cause of the outage is a bad mount entry.
Mounting the Filesystem and Chrooting In
Once you have a rescue shell, run journalctl -xb to look at the errors that stopped systemd from continuing the boot. If the journal points at a filesystem problem, unmount the suspect mount with umount /mnt/sysimage/var (or wherever the failure is) and run xfs_repair /dev/sdaN or fsck.ext4 /dev/sdaN depending on the filesystem. When the disk is clean, mount it back with mount /dev/sda2 /mnt/sysimage and chroot into the repaired tree using chroot /mnt/sysimage /bin/bash. From here, paths like /etc/fstab refer to the real root, and you can edit, run dracut, or invoke dnf without confusion.
Repairing fstab, initramfs, and dnf Rollbacks
Most boot failures on RHEL trace back to one of three things: a typo in /etc/fstab, a corrupted initramfs, or a kernel update that left the system in a bad state. Open /etc/fstab and confirm every UUID matches what blkid reports, paying close attention to the fs_freq and fs_passno columns. If a recent update is the culprit, you can roll back using dnf's history; the exact steps are laid out in this dnf rollback guide. To rebuild the initramfs after fixing fstab or installing new drivers, run dracut --force /boot/initramfs-$(uname -r).img $(uname -r).
Reinstalling the GRUB Bootloader
Sometimes the bootloader itself is the casualty, especially after a disk clone or a UEFI firmware update. Inside the chroot, reinstall GRUB on BIOS systems with grub2-install /dev/sda, and on UEFI systems with grub2-install --target=x86_64-efi --efi-directory /boot/efi --bootloader-id=rhel. Regenerate the configuration with grub2-mkconfig -o /boot/grub2/grub.cfg, exit the chroot, and run reboot. Watch for the familiar GRUB menu to reappear: if it does, the repair worked.
Hardening the Boot Process Against Future Failures
Once the server is healthy again, take a few minutes to harden the boot process. Set GRUB_TIMEOUT=5 in /etc/default/grub so you have time to interrupt the boot, and add nofail to experimental mount points in fstab so a missing share does not stall the system. Turn on persistent journaling with journalctl --vacuum-size=200M and forward logs to a central syslog server. Many Australian ops teams now run mobile dashboards alongside standard monitoring, and the reactive UI patterns described in this LiveData walkthrough are increasingly common in custom ops apps. Schedule the next change window for an arvo when the team is fresh, snapshot the root volume beforehand, and document every command you typed.
If the journal is clean and the system survives two reboots, the job is done. Rescue mode is not a last resort: it is the routine path back to a working system whenever RHEL refuses to boot, and treating it that way turns a stressful arvo into an ordinary Tuesday.