Keycloak Cluster on Three VMs: Engineers Detail Embedded Infinispan Setup, HA Testing Failures, and Bug Fix

A team deployed Keycloak for 25,000–30,000 users across a country using three isolated VMs instead of Kubernetes. They chose Embedded Infinispan with JGroups over an external cluster to reduce failure points and maintain session replication. PostgreSQL with Patroni handled shared storage and JDBC_PING discovery. Testing revealed that full datacenter outages triggered split-brain conditions and 5xx errors that the built-in healthcheck could not resolve. The engineers wrote a Python watcher script to detect multiple coordinators in the JGROUPS_PING table and restart affected containers. They reported the recovery bug to the Keycloak project, which was confirmed and fixed in a later release. The production cluster now survives single-DC loss without session loss.

HabrTranslated from Russian

Read at source

A project required centralized authentication and authorization for 25,000–30,000 active users distributed nationwide, demanding strong scalability and reliability. The team selected Keycloak and rejected a single instance because it could not meet throughput or SLA targets. They therefore built a three-node cluster on isolated virtual machines using Embedded Infinispan and JGroups.

Kubernetes was considered excessive for the environment. The chosen architecture placed each Keycloak node in a separate data center, shared a PostgreSQL database clustered with Patroni, and enabled cache mode ispn so sessions replicate across nodes. TCP ports 7800 and 57800 were opened between the VMs for Infinispan communication and failure detection.

Each node registers its address in the JGROUPS_PING table at startup, allowing dynamic membership without static TCPPING lists. The JVM flag -Djgroups.external_addr and explicit node-name entries in cache-ispn.xml ensured correct visibility of external IPs and hostnames across data centers.

Failover tests included node restarts, database connectivity loss, and complete data-center isolation. When only Keycloak nodes lost connectivity while the database remained reachable, the cluster recovered automatically. Full data-center outages, however, produced split-brain states, multiple coordinators, and persistent 5xx errors. The built-in healthcheck did not detect the condition even after an upgrade that added cluster-specific checks.

A temporary Python watcher was deployed beside each container. It polled health endpoints and the JGROUPS_PING table, restarting any node that appeared as an extra coordinator. The root cause was reported in GitHub issue 45980; Keycloak developers acknowledged the bug and shipped a fix. After verification on test and staging environments, the production cluster was updated and now tolerates single data-center failure without session loss.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Vulnerabilities & Exploits

Telegram for Android Patches Voice Call Vulnerability Enabling Code Execution
Yandex Patches Millions of Stations After eMudhra Certificate Keys Compromised
Cisco ASA and FTD Face DoS Vulnerability in SSL VPN Already Exploited in the Wild