I realize that I did not include what error I see. When my primary node is offline, the keycloak app could not access its database
2023-08-30 17:31:38,693 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (Timer-0) Acquisition timeout while waiting for new connection
2023-08-30 17:31:38,694 ERROR [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-0) Failed to run scheduled task ClearExpiredAdminEvents: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
so from there looked into the app configuration, and to my surprize I found that the keycloak app on the backup node is pointed to the primary node DB, which obviously was offline.
Specifically under the administrative settings of the app on the backup node, the settings of:
Defines the FQDN of the UCS instance used to change user password.
Database URI (e.g. jdbc:postgresql://dbhost/keycloak?ssl=require).
The interesting part is that there is a postgresql DB and relevant user, permission created on the backup node, but not used. Correcting these URIs resulted in broken keycloak install.
Tested to complete uninstall and reinstall, and found that the node installed second will have it DB pointed to the first app, i.e. if installed keycloak on the backup node first, and primary second, the app on the primary node would be pointed to the backup node.
If I specifically specified the URI during install, then I ended with a broken install, where install of app would not complete