Best Practice: Monitor additional services with the UCS-Dashboard

Scenario

Additional services shall be monitored graphically with the UCS Dashboard App.

Recommendation

The Dashboard is based on Prometheus and Grafana, therefore a service to be displayed needs to expose statistics as requested by Prometheus or you will need an additional Prometheus exporter per application.
If your service is emitting metrics, 3 steps are required to make them visible in the Dashboard:

Serve metrics via Apache

To gather the metrics with your Prometheus installation, they must be made available under [ADDRESS]/metrics-[SERVICE]/metrics/, for example https://dir-node.mydomain.intranet/metrics-ox/metrics/. This can be achieved with the following Apache Config:

LDAPTrustedMode TLS
<Location "/metrics-ox/metrics/">
        AuthName "Prometheus Access"
        AuthType Basic
        require valid-user
        <IfModule mod_authnz_ldap.c>
                AuthBasicProvider ldap
                AuthLDAPUrl "ldap://[LDAP-SERVER-NAME]:[LDAP-SERVER-PORT]/[LDAP-BASE]?uid?sub?(objectClass=*)"
                AuthLDAPBindDN [LDAP-HOSTDN]
                AuthLDAPBindPassword "exec:/bin/cat /etc/machine.secret"
        </IfModule>
        ProxyPass [METRICS-ENDPOINT] retry=0
        ProxyPassReverse [METRICS-ENDPOINT]
</Location>

where for OX the [METRICS-ENDPOINT] would be http://127.0.0.1:8009/metrics and [LDAP-SERVER-NAME], [LDAP-SERVER-PORT], [LDAP-BASE] and [LDAP-HOSTDN] can be gathered via ucr get + ldap/base, ldap/server/name, ldap/server/port and /ldap/hostdn.
Apache needs to be restarted afterwards: service apache2 reload

Add service to Prometheus custom targets

On the prometheus instance, the file /var/lib/univention-appcenter/apps/prometheus/conf/custom-targets.json must be created and your service needs to be included like so:

[
  {
    "targets": [
      "metrics-ox.dir-node.mydomain.intranet"
    ]
  }
]

afterwards the Dashboard Database needs to be restarted: univention-app restart prometheus

As for now (August 2022) Prometheus relabels both custom and automatically configured targets like this:

  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: ([^\.]+)\..*
    target_label: __metrics_path__
    replacement: /$1/metrics/
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: '[^\.]+\.(.*)'
    target_label: __address__
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace

The URL has to follow this format to be valid (e.g. in the example https://dir-node.mydomain.intranet/metrics-ox/metrics/ with scheme: https://A/B/C it has to be https://B.Ametrics-ox.dir-node.mydomain.intranet).

Make a new Dashboard

Since the Domain Dashboard and Server Dashboard cannot be edited, you need to create a new Dashboard in Grafana to be able to add a new graph. Inside this, your metrics can be queried with PromQL.

Further reading (optional)

Questions?

If you’re not sure whether the recommendations will fit into your scenario, please ask your Professional Services contact person, or create a new topic referencing this article.

5 Likes
Mastodon