JGroups

Keycloak Cluster on Three VMs: Engineers Detail Embedded Infinispan Setup, HA Testing Failures, and Bug Fix

A team deployed Keycloak for 25,000–30,000 users across a country using three isolated VMs instead of Kubernetes. They chose Embedded Infinispan with JGroups over an external cluster to reduce failure points and maintain session replication. PostgreSQL with Patroni handled shared storage and JDBC_PING discovery. Testing revealed that full datacenter outages triggered split-brain conditions and 5xx errors that the built-in healthcheck could not resolve. The engineers wrote a Python watcher script to detect multiple coordinators in the JGROUPS_PING table and restart affected containers. They reported the recovery bug to the Keycloak project, which was confirmed and fixed in a later release. The production cluster now survives single-DC loss without session loss.