PKI Storm: Managing 100,000 Simultaneous Certificate Requests in Kubernetes Recovery Scenarios

A large organization's PKI infrastructure faced a critical bottleneck when a data center outage triggered simultaneous startup of tens of thousands of Kubernetes pods, each requiring mTLS certificates. The existing setup using ESAUS and Citadel routed all requests through external certificate authorities that could only sustain 50-70 RPS against an incoming burst of 100,000 requests. Average daily load of 10-11 RPS had masked the thundering herd risk during mass recovery. Scaling the CA 15x was rejected due to cost and the fundamental dependency on real-time signing. The team introduced pre-issuance of certificates stored in a dedicated Unified Secret Storage (ЕХС) layer that supports 14,000 RPS reads while the CA continues normal operation. This architectural separation of issuance and consumption reduced recovery time from nearly 24 minutes to seconds while shifting focus to secure secret lifecycle management including KRA key protection.

HabrTranslated from Russian

Read at source

In large enterprises, TLS certificates are required everywhere: for services, workstations, containers, internal APIs, and system integrations. When thousands of consumers exist, PKI must handle massive request flows rather than individual certificates. One organization processed approximately 320 million certificate issuance requests per year.

Automation relied on the ESAUS central management system, which validates requests against security policies and forwards them to external certificate authorities without issuing certificates itself. For Kubernetes with Istio, the Citadel component of ESAUS was configured as the certificate source instead of Istio’s built-in CA. Citadel validates requests and context before passing them through ESAUS to corporate CAs, integrating Kubernetes into the enterprise PKI.

This created a dependency: Kubernetes recovery speed became tied to external CA performance. After a data center outage, dozens of thousands of pods restarted simultaneously. Because services used mTLS, each pod needed its own certificate and private key. In normal operation requests were spread over time, but during recovery roughly 100,000 requests arrived in a short window.

At 70 RPS the theoretical processing time reached almost 24 minutes. Timeouts triggered retries, amplifying load in a thundering herd effect. The infrastructure recovered faster than the CA could issue certificates, revealing that average load figures (10–11 RPS) did not predict burst behavior.

Vertical or horizontal CA scaling was evaluated. Reaching 1,050 RPS would theoretically cut storm time to 1.5 minutes, yet such capacity would remain idle more than 99 % of the time. The team instead decoupled issuance from consumption by pre-generating certificates and keys during normal operation and storing them in a secure reserve.

The reserve resides in the new Unified Secret Storage (ЕХС) layer, which provides centralized protected storage, strict authentication, access controls, auditing, high read throughput, and object lifecycle management. In performance mode, ЕХС delivers up to 14,000 RPS for existing secrets from standby nodes, allowing the burst to be served from reads rather than real-time CA operations.

The approach also addressed Key Recovery Agent (KRA) protection by keeping the KRA private key outside the CA and releasing it only for approved recovery operations. Over time the same storage layer absorbed additional operational secrets beyond the emergency reserve, evolving into a broader corporate secret management platform.

Key lessons include recognizing that average load poorly predicts disaster behavior, that some bottlenecks are better solved by separating production and consumption than by scaling, and that every architectural mitigation transfers complexity rather than eliminating it.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Other

Mail.ru Reduces Lifetime 1TB Cloud Storage Promo to Standard 8GB Quota for Select Users
Kimi K3 Open Model Surpasses Closed AI Leaders with Million-Token Context and 896 Experts
Innostage Completes Functional Testing of PT NGFW 1.10.3 Across 156 Criteria