Public prerelease · version 0.1.0-alpha.9

Fix broken Linux servers without risking a real one.

A disposable systemd server with thirteen realistic incident tickets, ordered hints and outcome-based verification. Choose Ubuntu, Debian or Rocky Linux and practise the evidence-first repair loop. Free and open source under the MIT licence.

Launch in Codespaces Quick start

The systemd host uses a privileged container. Read the safety boundary before running it locally.

The rescue loop

A server gives you evidence before it gives you answers

Start from the incident ticket, inspect the live system and make the smallest safe change. The verifier checks the recovered outcome, not whether you copied one expected command.

01
Break
Apply one known incident. The lab refuses to stack a second fault over the first.
02
Investigate
Enter relay, collect evidence and narrow the failing layer before changing it.
03
Repair
Restore the intended behaviour without disabling the protection around it.
04
Prove
Leave the host and run the outcome-based verifier. Resume later with down and up, or reset when you want a clean run.

bash

./lab up ubuntu
./lab break 01
./lab shell
./lab verify 01
./lab reset

Choose your target

One incident, three current Linux user spaces

Run one distribution at a time and see which evidence is universal and where package-family details differ.

./lab up ubuntu

Ubuntu 26.04 LTS

apt package family

./lab up debian

Debian 13

apt package family

./lab up rocky

Rocky Linux 10

dnf package family

Available incidents

Thirteen complete incidents in the public prerelease

Each one includes a ticket, three ordered hints, a spoiler-fenced solution and a verifier that never mutates the server.

Incident 01

Service failure

Trace a systemd service trapped in a restart loop, find the bad override and restore the healthy service without weakening its restart policy.

systemctl
journalctl
unit overrides

Incident 02

Full filesystem

Follow an application write failure to the correct filesystem, measure what consumed it and recover capacity without moving the data.

df
findmnt
du

Incident 03

DNS ghost

Find a stale local name-service override that shadows the lab network's DNS answer, restore the intended resolution path and recover the upstream systemd check.

getent
dig
nsswitch

Incident 04

Permission denied

Restore least-privilege write access after maintenance changes application-data ownership, without running the service as root or making the directory world-writable.

namei
stat
ownership and modes

Incident 05

Runaway process

Trace sustained CPU use to a restart-managed worker, identify its owning systemd unit and stop it from returning without killing an unexplained PID.

ps
/proc
systemctl

Incident 06

Invalid configuration

Follow a service failure to malformed JSON, validate the deployed file and atomically restore the last-known-good configuration already on the host.

journalctl
json.tool
atomic rollback

Incident 07

Wrong listener

Explain why an in-container health check passes while the published service fails, then restore the listener to a container-network address.

ip
ss
curl

Incident 08

Upstream port

Trace a failed systemd probe to the wrong external port and restore its healthy internal upstream endpoint without exposing or changing the companion service.

systemctl
getent
curl

Incident 09

Port conflict

Find the unauthorised listener already holding the application port, remove it from this boot and the next, and restore the service without changing its published port.

ss
systemctl
journalctl

Incident 10

Scheduled regression

Explain why a known-good configuration keeps reverting, stop the unauthorised timer restoring it, and make the correct deployment survive a restart.

systemd timers
journalctl
diff

Incident 11

Deleted open file

Recover filesystem space a directory scan cannot account for, trace the unlinked file to the process still holding it and release the bytes without rebooting.

df
/proc/*/fd
systemctl

Incident 12

Inode exhaustion

Explain a no-space error on a filesystem with free blocks, identify the capacity that is actually exhausted and remove only the obsolete artefacts responsible.

df -i
find
journalctl

Incident 13

Backup sprawl

Trace a full data filesystem to a local backup schedule with no retention, preserve a recent complete archive, apply a safe retention response and stop the fault from returning.

du
systemd timers
retention

What makes it practical

Real evidence, disposable consequences

Tickets, not toy prompts

Each incident starts with a short operational ticket. The server exposes the evidence, and more than one safe repair can pass.

A real systemd target

Inspect genuine units, process state and journal entries on a Linux host instead of reading invented terminal output.

Three Linux families

Repeat the same incident on current Ubuntu, Debian and RHEL-compatible Rocky Linux user spaces, one target at a time.

Outcome-based verification

The verifier checks whether the service is healthy again. It does not demand a memorised command sequence or reveal the answer.

Disposable state you can resume

Down and up preserve an active incident for that distribution. Reset removes the saved fault and rebuilds a known-good server when you want a clean run.

Browser or local Docker

Launch a private Codespace with a dedicated Docker daemon or use the matching Bash and PowerShell wrappers on your own machine.

Capability boundary

Honest about what Docker cannot teach

This track teaches service, log, process, filesystem, permission, DNS, configuration and networking diagnosis in real Linux user spaces. Containers share the Docker host kernel, so it does not pretend to teach GRUB, initramfs, kernel selection, physical disks or an actual machine's boot path.

Read the full FAQ →

Start with the method

Linux rescue starts with evidence, not commands

The introductory article explains the five-step incident method behind every drill and the signals worth collecting before you touch a broken server.

Read the introduction View source