systemd

Prod is Down: Free gVisor-Based Simulator Gives Strangers Root Access for Realistic Linux Incident Training

A Russian systems administrator has launched Prod is Down, a free online platform that lets users practice real incident response by granting them root consoles on deliberately broken Linux servers. The tool currently offers 13 scenarios covering PostgreSQL failures, OOM Killer events, full disks, expired certificates, nginx 502 errors, inode exhaustion, DNS issues, and open file limits. Each exercise runs inside gVisor sandboxes with strict resource limits, internal-only networking, and automated verification that checks service health rather than player commands. The author implemented an emulated systemd in Python so trainees can use familiar systemctl and journalctl commands without running a real init system. Multiple layers of isolation, including overlay filesystem tweaks and size monitoring, prevent abuse while still allowing sudo and realistic debugging. The platform records three random variants per scenario so memorized fixes from previous runs do not work on subsequent attempts.

What Server Logs Reveal About Attackers: Analyzing Suspicious Requests on Linux

Server logs provide critical insights into potential intrusions by recording events from the operating system, SSH authentication, and web applications. The guide explains how to examine Linux system journals with journalctl, review authentication logs on Debian and RHEL systems, and inspect Nginx access logs for unusual HTTP requests. Analysts learn to identify patterns such as rapid failed login attempts using multiple usernames from a single IP address, which may indicate brute-force or automated scanning activity. The article also covers counting the most active IP addresses with awk and sort commands while stressing the need to compare findings against normal traffic baselines. Additional recommendations include logging structured security events in JSON format with fields for timestamp, user ID, resource, and outcome, following OWASP guidelines. The text warns against storing sensitive data like passwords in logs and highlights limitations such as possible log tampering or unsynchronized server clocks. A practical seven-step investigation plan helps responders preserve evidence and verify hypotheses before taking action.

Deploying Self-Hosted Hysteria 2 Proxy on Debian-Based Linux VPS via Terminal

A detailed guide explains how to set up a personal Hysteria 2 proxy server on a KVM VPS running Debian or Ubuntu without any web panels. The process begins with generating ed25519 SSH keys, hardening the sshd_config file, and restricting access with ufw to only TCP port 22 and UDP port 443. Hysteria 2 is downloaded from GitHub, made executable, and configured using a TOML file that enables salamander obfuscation and a self-signed TLS certificate. A custom systemd unit ensures the service restarts on failure. The client configuration includes SHA256 pinning of the server certificate to prevent MITM attacks. The guide emphasizes manual CLI operations that apply equally to other services such as Nginx and stresses checking local laws before deployment.

Secure Custom Domain Setup for Client Status Pages Using CNAME, Certbot and Go Instead of ACME On-Demand

A detailed case study describes how a monitoring service implemented white-label status pages on customer domains without relying on ACME on-demand certificate issuance. The approach uses pre-validated domains stored in a database, background DNS checks via CNAME or A-record matching, and a root helper script running on a systemd timer to handle certbot issuance and nginx configuration. Key design choices separate privileges so the Go application never touches certificates or nginx directly, while loopback endpoints are protected against proxy header spoofing. The solution explicitly addresses risks such as DoS through malicious Host headers exhausting Let's Encrypt rate limits, private key exposure, and first-visitor latency during TLS handshakes. Hysteresis in DNS status prevents temporary resolution glitches from disabling active customer pages. The entire implementation stays under a few hundred lines of code and runs reliably on a single server with nginx in front of the Go application.