Prod is Down: Free gVisor-Based Simulator Gives Strangers Root Access for Realistic Linux Incident Training

A Russian systems administrator has launched Prod is Down, a free online platform that lets users practice real incident response by granting them root consoles on deliberately broken Linux servers. The tool currently offers 13 scenarios covering PostgreSQL failures, OOM Killer events, full disks, expired certificates, nginx 502 errors, inode exhaustion, DNS issues, and open file limits. Each exercise runs inside gVisor sandboxes with strict resource limits, internal-only networking, and automated verification that checks service health rather than player commands. The author implemented an emulated systemd in Python so trainees can use familiar systemctl and journalctl commands without running a real init system. Multiple layers of isolation, including overlay filesystem tweaks and size monitoring, prevent abuse while still allowing sudo and realistic debugging. The platform records three random variants per scenario so memorized fixes from previous runs do not work on subsequent attempts.

HabrTranslated from Russian

Read at source

A systems administrator with four years of experience has released Prod is Down, a free browser-based trainer that lets users receive root access to deliberately broken Linux servers and practice incident response without risking production systems.

The platform currently contains 13 scenarios. After a short briefing that includes recent changes and symptoms, the server begins to fail while metrics are collected. Players then open a live terminal, investigate with standard tools such as top, df, journalctl, and ss, and finally press a verification button that runs an external health-check script.

Security relies on gVisor (runsc) instead of runc, per-container Docker networks marked internal, removal of most Linux capabilities, strict memory and process limits, and a watchdog that deletes containers exceeding 1 GB of writes. The author also added --allow-suid and --overlay2=none flags after discovering that default gVisor settings broke sudo and disk accounting.

Because players have root inside the container, verification never trusts files or binaries present in the environment. The check script is supplied at runtime from the host, and any modification to /usr, /bin, /etc/sudoers or scenario-specific files causes immediate rejection. Scenario parameters are stored as a SHA-256 hash so players cannot alter the failure conditions.

An emulated systemd written in Python provides realistic behavior for systemctl, journalctl, Restart=, EnvironmentFile=, and LimitNOFILE= so that experience transfers directly to real servers that run full systemd.

Early user feedback prompted interface changes: the “Take incident” button was moved to a fixed position, and a hint prompt now appears after three failed verification attempts. The first scenario remains open without registration.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Other

Mail.ru Reduces Lifetime 1TB Cloud Storage Promo to Standard 8GB Quota for Select Users
Kimi K3 Open Model Surpasses Closed AI Leaders with Million-Token Context and 896 Experts
Innostage Completes Functional Testing of PT NGFW 1.10.3 Across 156 Criteria