A project required centralized authentication and authorization for 25,000–30,000 active users distributed nationwide, demanding strong scalability and reliability. The team selected Keycloak and rejected a single instance because it could not meet throughput or SLA targets. They therefore built a three-node cluster on isolated virtual machines using Embedded Infinispan and JGroups.
Kubernetes was considered excessive for the environment. The chosen architecture placed each Keycloak node in a separate data center, shared a PostgreSQL database clustered with Patroni, and enabled cache mode ispn so sessions replicate across nodes. TCP ports 7800 and 57800 were opened between the VMs for Infinispan communication and failure detection.
Each node registers its address in the JGROUPS_PING table at startup, allowing dynamic membership without static TCPPING lists. The JVM flag -Djgroups.external_addr and explicit node-name entries in cache-ispn.xml ensured correct visibility of external IPs and hostnames across data centers.
Failover tests included node restarts, database connectivity loss, and complete data-center isolation. When only Keycloak nodes lost connectivity while the database remained reachable, the cluster recovered automatically. Full data-center outages, however, produced split-brain states, multiple coordinators, and persistent 5xx errors. The built-in healthcheck did not detect the condition even after an upgrade that added cluster-specific checks.
A temporary Python watcher was deployed beside each container. It polled health endpoints and the JGROUPS_PING table, restarting any node that appeared as an extra coordinator. The root cause was reported in GitHub issue 45980; Keycloak developers acknowledged the bug and shipped a fix. After verification on test and staging environments, the production cluster was updated and now tolerates single data-center failure without session loss.


