HowTo: Get prometheus data from a server and import it into an other

Knowledge Base Community
, , ,
1

HowTo: Transferring Historical Prometheus Data for Multiple UCS Servers

This procedure exports all Prometheus metrics for a selected group of UCS servers over one common time range and imports them into another UCS Dashboard Database. The imported historical data can then be viewed in the UCS Dashboard on the destination system.

Scope

  • Source system: UCS server running the UCS Dashboard Database app (prometheus)
  • Destination system: UCS server running the target UCS Dashboard Database app (prometheus)
  • Prometheus version: 2.53.3 on both systems
  • Required user: root
  • Transfer format: OpenMetrics
  • Example: Six servers using the same time range

This procedure transfers historical metrics only. It does not register the imported servers in UCS and does not configure them as active Prometheus scrape targets on the destination system.

Important Considerations

  • The requested data must still be present on the source system.
  • The default Prometheus retention period is 15 days.
  • The end of the exported interval must be at least three hours in the past. Prometheus does not consider backfilling into its current, mutable head block safe.
  • Do not import the same file more than once. Repeated imports create duplicate or overlapping TSDB blocks.
  • Imported blocks are subject to the retention policy of the destination system and may be removed immediately if they are too old.
  • Metric names, labels, values, and timestamps of regular float samples are preserved.
  • Native histograms and staleness markers cannot be represented by this OpenMetrics transfer.
  • Use the exact Prometheus instance label values, including full domain names where applicable.
  • One combined export file is appropriate when all selected servers use the same time range.

1. Verify the Prometheus Version

Run on both the source and destination systems:

univention-app shell prometheus prometheus --version

The expected version is 2.53.3.

2. Define the Time Range

Run on the source system. Use UTC timestamps in RFC 3339 format:

START='2026-08-12T10:00:01Z'
END='2026-08-12T16:00:00Z'

Convert the timestamps to Unix timestamps in milliseconds, as required by promtool tsdb dump-openmetrics:

START_MS="$(date --date="$START" +%s%3N)"
END_MS="$(date --date="$END" +%s%3N)"

printf 'Start: %s (%s)\nEnd:   %s (%s)\n' \
  "$START" "$START_MS" "$END" "$END_MS"

Note: Verify that END is at least three hours in the past before continuing.

3. Identify the Instance Labels

Run on the source system:

univention-app shell prometheus \
  promtool query series \
  --start "$START" \
  --end "$END" \
  --match 'node_boot_time_seconds' \
  http://localhost:9090/metrics-prometheus/

This command runs inside the Prometheus app and prints the matching series, including their instance labels. Use the instance values exactly as displayed.

You can also use following host-side API request, which may not work on installations where Prometheus port 9090 is not exposed on the host loopback interface:

curl --silent --show-error --fail-with-body --get \
  'http://127.0.0.1:9090/metrics-prometheus/api/v1/series' \
  --data-urlencode 'match[]=node_boot_time_seconds' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END"

Use the univention-app shell prometheus promtool query series command above in that case.

4. Configure the Selected Servers

Run on the source system and replace the example names with the exact instance values:

SERVERS=(
  'server1.example.com'
  'server2.example.com'
  'server3.example.com'
  'server4.example.com'
  'server5.example.com'
  'server6.example.com'
)

Build one Prometheus series selector per server:

MATCH_ARGS=()

for SERVER in "${SERVERS[@]}"; do
  MATCH_ARGS+=(--match "{instance=\"$SERVER\"}")
done

Multiple --match arguments select the union of all matching series. The resulting file therefore contains all metrics carrying one of the selected instance labels.

5. Export the Historical Data

Run on the source system:

EXPORT_FILE="/root/prometheus-selected-servers-${START_MS}-${END_MS}.openmetrics"

univention-app shell prometheus \
  promtool tsdb dump-openmetrics \
  --min-time "$START_MS" \
  --max-time "$END_MS" \
  "${MATCH_ARGS[@]}" \
  /prometheus/data \
  > "$EXPORT_FILE"

Verify the result:

test -s "$EXPORT_FILE" && ls -lh "$EXPORT_FILE"

Metrics without an instance label are not included. This is intentional for a server-specific export.

6. Transfer the Export File

Run on the source system and replace the destination name:

DESTINATION='dc01.example.com'

scp "$EXPORT_FILE" "root@${DESTINATION}:/root/"

Display the transferred filename for use on the destination system:

basename "$EXPORT_FILE"

7. Verify Destination Retention

Run on the destination system:

ucr get prometheus/storage/tsdb/retention

Check the effective runtime setting:

curl --silent --show-error --fail-with-body \
  'http://127.0.0.1:9090/metrics-prometheus/api/v1/status/flags' |
jq -r '.data["storage.tsdb.retention.time"]'

The retention period must cover the age of the imported data. Otherwise, Prometheus may remove the imported blocks automatically.

8. Prepare the Import File

Run on the destination system. Set the exact filename transferred in step 6:

EXPORT_FILE='/root/prometheus-selected-servers-START_MS-END_MS.openmetrics'

Replace START_MS and END_MS with the values in the actual filename, then verify it:

test -s "$EXPORT_FILE" && ls -lh "$EXPORT_FILE"

Copy it into the configuration directory mounted into the Prometheus container:

install \
  --owner=65534 \
  --group=0 \
  --mode=640 \
  "$EXPORT_FILE" \
  /var/lib/univention-appcenter/apps/prometheus/conf/prometheus-import.openmetrics

The file is now available inside the container as /etc/prometheus/prometheus-import.openmetrics.

9. Import the Data

Run once on the destination system:

univention-app shell prometheus \
  promtool tsdb create-blocks-from openmetrics \
  /etc/prometheus/prometheus-import.openmetrics \
  /prometheus/data

A successful import prints a table containing block ULIDs, minimum and maximum timestamps, sample counts, series counts, and block sizes.

Do not run this command again for the same export file.

Restart Prometheus:

univention-app restart prometheus

Wait until Prometheus is ready:

until curl --silent --fail \
  'http://127.0.0.1:9090/metrics-prometheus/-/ready' >/dev/null
do
  sleep 2
done

10. Verify the Imported Blocks

Run on the destination system:

univention-app shell prometheus \
  promtool tsdb list --human-readable /prometheus/data

The displayed minimum and maximum times must cover the imported interval.

11. Verify the Imported Servers

Run on the destination system, using the same RFC 3339 time range as during export:

START='2026-08-12T10:00:01Z'
END='2026-08-12T16:00:00Z'

univention-app shell prometheus \
  promtool query series \
  --start "$START" \
  --end "$END" \
  --match 'node_boot_time_seconds' \
  http://localhost:9090/metrics-prometheus/

The output should contain the instance labels of all imported servers.

Optionally verify actual metric values for one server:

SERVER='server1.example.com'

curl --silent --show-error --fail-with-body --get \
  'http://127.0.0.1:9090/metrics-prometheus/api/v1/query_range' \
  --data-urlencode "query=node_load1{instance=\"$SERVER\"}" \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode 'step=1m' |
jq '.data.result'

If host port 9090 is unavailable, use the block and series checks above instead.

12. View the Data in UCS Dashboard

Open the UCS Dashboard on the destination system:

https://DESTINATION-SYSTEM/ucs-dashboard/

Then:

  1. Open Server Dashboard.
  2. Select an absolute time range matching START and END.
  3. Reload the dashboard.
  4. Open the server selector.
  5. Select one of the imported servers.

The server selector is derived from node_boot_time_seconds series in the currently selected time range. Imported historical servers therefore do not appear when Grafana is set to a range such as Last 24 hours that does not include the imported data.

13. Remove the Transfer Files

After successful verification, run on the destination system:

rm -f \
  /var/lib/univention-appcenter/apps/prometheus/conf/prometheus-import.openmetrics \
  "$EXPORT_FILE"

This removes only the transferred OpenMetrics files. The imported TSDB blocks remain in Prometheus until removed by its retention policy.