Problem
The intercom-service pods were repeatedly killed because of out-of-memory conditions.
Initially, the pods were configured with 128MiB of RAM. The memory was then increased to 256MiB, but the pods were also killed with this configuration. The memory was then raised to 512MiB.
The environment has approximately 2,000 users actively using the intercom-service.
Questions raised:
- What are the sizing recommendations for the
intercom-service?
Environment
Component: intercom-service
Runtime: Node.js 18.20.x
Platform: Kubernetes
Memory configuration observed: 128MiB, 256MiB, 512MiB
Scale: Approximately 2,000 active users using the intercom-service
Root Cause
There is an open bug regarding the scaling requirements of the intercom-service:
Bug Report: Forge Bug #58660
The priority of the bug was raised because this topic is relevant for the documentation.
Investigation
Internally, the following specifications regarding the intercom-service are communicated:
- First, it should be attempted to raise the memory limit to
4GiB. - Second, the number of replicas can also be raised. The service is scalable.
- If that does not help, that would indicate a memory leak and one should attempt to roll back to the previous version.
Solution
Raise the memory limit of the intercom-service to 4GiB.
If that does not help, increase the number of replicas.
If increasing the memory limit and the number of replicas does not help, this would indicate a memory leak. In that case, attempt to roll back to the previous version.