Problem: Prometheus storage size unexpectedly large

Problem

The Prometheus app on the Primary Domain Controller (DC) is using a significant amount of disk space:

# du -shx /var/lib/univention-appcenter/apps/prometheus/data
22G     /var/lib/univention-appcenter/apps/prometheus/data

Is a 22 GB Prometheus data directory normal? Can the storage usage be optimized?


Root Cause

Prometheus stores time-series data locally on disk. The storage size depends heavily on:

  • Number of monitored hosts or services
  • Scraping interval (how often metrics are collected)
  • Data retention period (how long metrics are kept)

A 22 GB data directory is not unusual in environments where multiple hosts are being monitored or where detailed metrics are collected at frequent intervals.
In smaller setups, typical usage ranges between 1–5 GB, but this increases quickly with larger domains or shorter scrape intervals.


Solution

To optimize storage usage, you can adjust Prometheus’ configuration parameters directly through the Univention App Center.

Run the following command to list available options:

univention-app configure prometheus --list

Key parameters to review:

  • prometheus/storage/tsdb/retention
    Defines how long Prometheus keeps historical data (default: 15d).
    Reducing this value will decrease disk usage.

  • scrape/interval
    Controls how often metrics are collected (default: 1m).
    Increasing this interval (e.g., to 2m or 5m) can significantly reduce data volume.

  • prometheus/generic/server/parameters
    Allows passing additional flags to Prometheus.
    You can limit disk usage by setting a fixed maximum size, for example:

    --storage.tsdb.retention.size=10GB
    

    This will cap Prometheus’ storage to 10 GB regardless of the retention time.

For a detailed explanation of how retention settings and size limits interact, see the following article:
:point_right: Configuring Prometheus storage retention (robustperception.io)


See also:

This topic was automatically closed after 24 hours. New replies are no longer allowed.