runc

Prod is Down: Free gVisor-Based Simulator Gives Strangers Root Access for Realistic Linux Incident Training

A Russian systems administrator has launched Prod is Down, a free online platform that lets users practice real incident response by granting them root consoles on deliberately broken Linux servers. The tool currently offers 13 scenarios covering PostgreSQL failures, OOM Killer events, full disks, expired certificates, nginx 502 errors, inode exhaustion, DNS issues, and open file limits. Each exercise runs inside gVisor sandboxes with strict resource limits, internal-only networking, and automated verification that checks service health rather than player commands. The author implemented an emulated systemd in Python so trainees can use familiar systemctl and journalctl commands without running a real init system. Multiple layers of isolation, including overlay filesystem tweaks and size monitoring, prevent abuse while still allowing sudo and realistic debugging. The platform records three random variants per scenario so memorized fixes from previous runs do not work on subsequent attempts.