Prometheus not running after upgrading prometheus to 2.35.0-3

 60184 actions.start                    22-07-19 19:16:23 [    INFO]: Starting docker-app-prometheus (via systemctl): docker-app-prometheus.service.
 60184 actions.start                    22-07-19 19:16:23 [   DEBUG]: /etc/init.d/docker-app-prometheus returned with 0
 60184 actions.start.progress           22-07-19 19:16:23 [   DEBUG]: 100
 60184 docker.base.container.f7a5       22-07-19 19:16:26 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 docker.base.container.f7a5       22-07-19 19:16:26 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 rm -f /etc/timezone /etc/localtime
 60184 docker.base                      22-07-19 19:16:26 [    INFO]: Running command: docker cp /etc/timezone f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/timezone
 60184 docker.base                      22-07-19 19:16:27 [    INFO]: Running command: docker cp /etc/localtime f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/localtime
 60184 actions.configure                22-07-19 19:16:28 [   DEBUG]: Calling configure
 60184 actions.configure.progress       22-07-19 19:16:28 [   DEBUG]: 0
 60184 actions.configure                22-07-19 19:16:28 [    INFO]: Configuring prometheus=2.35.0-3
 60184 settings                         22-07-19 19:16:28 [    INFO]: Setting insecure/skip/verify to 'false'
 60184 settings                         22-07-19 19:16:28 [    INFO]: Setting prometheus/storage/tsdb/retention to '15d'
 60184 settings                         22-07-19 19:16:28 [    INFO]: Setting prometheus/query/lookback-delta to '12h'
 60184 settings                         22-07-19 19:16:28 [    INFO]: Unsetting prometheus/generic/server/parameters
 60184 settings                         22-07-19 19:16:28 [    INFO]: Setting scrape/interval to '1m'
 60184 docker.configure.container.f7a5  22-07-19 19:16:28 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 docker.configure.container.f7a5  22-07-19 19:16:28 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 which ucr
 60184 actions.configure                22-07-19 19:16:28 [ WARNING]: ucr cannot be found, falling back to changing the database file directly
 60184 actions.configure.progress       22-07-19 19:16:33 [   DEBUG]: 100
 60184 actions.upgrade.progress         22-07-19 19:16:33 [   DEBUG]: 50
 60184 actions.upgrade                  22-07-19 19:16:33 [    INFO]: Executing interface restore_data_before_setup for prometheus
 60184 actions.upgrade                  22-07-19 19:16:33 [    INFO]: No interface defined
 60184 actions.upgrade                  22-07-19 19:16:33 [    INFO]: Executing interface restore_data_after_setup for prometheus
 60184 actions.upgrade                  22-07-19 19:16:33 [    INFO]: No interface defined
 60184 actions.update-certificates      22-07-19 19:16:33 [   DEBUG]: Calling update-certificates
 60184 actions.update-certificates.progress 22-07-19 19:16:33 [   DEBUG]: 0
 60184 actions.update-certificates      22-07-19 19:16:33 [    INFO]: updating certificates for prometheus=2.35.0-3
 60184 update-certificates.container.f7a5 22-07-19 19:16:33 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:33 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 mkdir -p /usr/local/share/ca-certificates
 60184 update-certificates.container.f7a5 22-07-19 19:16:35 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates              22-07-19 19:16:35 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsCA/CAcert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/usr/local/share/ca-certificates/ucs.crt
 60184 update-certificates              22-07-19 19:16:35 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/usr/local/share/ca-certificates
 60184 update-certificates              22-07-19 19:16:35 [   ERROR]: Command docker cp /etc/univention/ssl/ucsCA/CAcert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/usr/local/share/ca-certificates/ucs.crt failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/usr/local/share/ca-certificates (1)
 60184 update-certificates.container.f7a5 22-07-19 19:16:35 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:35 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 which update-ca-certificates
 60184 update-certificates.container.f7a5 22-07-19 19:16:35 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 mkdir -p /etc/univention/ssl/ucsCA/
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates              22-07-19 19:16:36 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsCA/CAcert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsCA/CAcert.pem
 60184 update-certificates              22-07-19 19:16:36 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsCA
 60184 update-certificates              22-07-19 19:16:36 [   ERROR]: Command docker cp /etc/univention/ssl/ucsCA/CAcert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsCA/CAcert.pem failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsCA (1)
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 mkdir -p /etc/univention/ssl/docker-host-certificate
 60184 update-certificates.container.f7a5 22-07-19 19:16:36 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates.container.f7a5 22-07-19 19:16:37 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:37 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 chmod 750 /etc/univention/ssl/docker-host-certificate
 60184 update-certificates.container.f7a5 22-07-19 19:16:37 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates              22-07-19 19:16:37 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsserver.top2.top1/cert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate/cert.perm
 60184 update-certificates              22-07-19 19:16:37 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate
 60184 update-certificates              22-07-19 19:16:37 [   ERROR]: Command docker cp /etc/univention/ssl/ucsserver.top2.top1/cert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate/cert.perm failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate (1)
 60184 update-certificates              22-07-19 19:16:37 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsserver.top2.top1/private.key f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate/private.key
 60184 update-certificates              22-07-19 19:16:38 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate
 60184 update-certificates              22-07-19 19:16:38 [   ERROR]: Command docker cp /etc/univention/ssl/ucsserver.top2.top1/private.key f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate/private.key failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/docker-host-certificate (1)
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 mkdir -p /etc/univention/ssl/ucsserver.top2.top1
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [   DEBUG]: Using container.f7a5 for container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [   DEBUG]: Calling docker exec -u root -it f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 chmod 750 /etc/univention/ssl/ucsserver.top2.top1
 60184 update-certificates.container.f7a5 22-07-19 19:16:38 [ WARNING]: Error response from daemon: Container f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6 is not running
 60184 update-certificates              22-07-19 19:16:38 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsserver.top2.top1/cert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1/cert.perm
 60184 update-certificates              22-07-19 19:16:39 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1
 60184 update-certificates              22-07-19 19:16:39 [   ERROR]: Command docker cp /etc/univention/ssl/ucsserver.top2.top1/cert.pem f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1/cert.perm failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1 (1)
 60184 update-certificates              22-07-19 19:16:39 [    INFO]: Running command: docker cp /etc/univention/ssl/ucsserver.top2.top1/private.key f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1/private.key
 60184 update-certificates              22-07-19 19:16:40 [    INFO]: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1
 60184 update-certificates              22-07-19 19:16:40 [   ERROR]: Command docker cp /etc/univention/ssl/ucsserver.top2.top1/private.key f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1/private.key failed with: Error: No such container:path: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6:/etc/univention/ssl/ucsserver.top2.top1 (1)
 60184 actions.update-certificates      22-07-19 19:16:40 [   DEBUG]: Calling /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/prometheus_20220530120047.update_certificates --version 2.35.0-3 --error-file /tmp/tmpQ90NUE
 60184 actions.update-certificates      22-07-19 19:16:43 [    INFO]: f7a56728ebef7e6cd68207d468fd5058296ab2850df361259bc7971f4f5ec9a6
 60184 actions.update-certificates      22-07-19 19:16:43 [   DEBUG]: /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/prometheus_20220530120047.update_certificates returned with 0
 60184 actions.update-certificates.progress 22-07-19 19:16:43 [   DEBUG]: 100
 60184 actions.upgrade.progress         22-07-19 19:16:43 [   DEBUG]: 80
 60184 actions.upgrade                  22-07-19 19:16:43 [    INFO]: Installing join script /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/prometheus_20220530120047.inst

ts=2022-07-19T17:35:37.995Z caller=main.go:1188 level=error msg="Failed to apply configuration" err="unable to load specified CA cert /usr/local/share/ca-certificates/ucs.crt: open /usr/local/share/ca-certificates/ucs.crt: no such file or directory"

We are working on a fix. A workaround which should bring the app back to work should be:

sed -i 's#ca_file: .*#ca_file: /etc/ssl/certs/ca-certificates.crt#g' /var/lib/univention-appcenter/apps/prometheus/conf/prometheus.yml
univention-app restart prometheus
1 Like
Mastodon