Upgrade to UCS 4.4-2 fails caused by package docker.io

SirTux · September 24, 2019, 4:49pm

Hi,

unluckily the upgrade to UCS 4.4-2 fails because the setup of docker.io fails:

Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458334719+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458381042+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458416376+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458428147+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458501798+02:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458511016+02:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458540141+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458525655+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458589325+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4209fc0f0, CONNECTING" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.458607391+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420189c30, CONNECTING" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.459224698+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420189c30, READY" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.459528574+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4209fc0f0, READY" module=grpc
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.459774586+02:00" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
Sep 24 18:44:31 ucsmaster dockerd[46022]: time="2019-09-24T18:44:31.459815978+02:00" level=error msg="[graphdriver] prior storage driver aufs failed: driver not supported"
Sep 24 18:44:31 ucsmaster dockerd[46022]: Error starting daemon: error initializing graphdriver: driver not supported
Sep 24 18:44:31 ucsmaster systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 18:44:31 ucsmaster systemd[1]: Failed to start Docker Application Container Engine.

It seems to be an issue with aufs … autsch

Best,
SirTux

SirTux · September 24, 2019, 5:06pm

As workarround I’ve created the file /etc/docker/daemon.json:

{
          "storage-driver": "overlay"
}

I don’t know if this the correct value. Luckily I’ve no docker apps installed on the DC Master.

EDIT: Should be correct. At least

docker info

ran on non-updated host was saying the same.

zash · September 24, 2019, 6:03pm

Same here,

applying the solution listed above (create file and content) does its job also here. Docker is running again

Thanks,

I always sent the error report to Univention-Support

SirTux · September 24, 2019, 6:11pm

I’m glad that this helped also you. I hope you haven’t forgotton to run

dpkg --configure -a
univention-upgrade

afterwards.

Seems that this bug only occurs if there exist the path /var/lib/docker/aufs. This seems to be the case only on older systems. That explains that this bug wasn’t found during QA process.

zash · September 24, 2019, 7:52pm

Yes,

i have run both after creating the file. So far docker and its apps are running and the upgrade to 4.4-2 is running fine afterwards!

/var/lib/docker/aufs contains only three empty subfolders here

I think Univention will rework on its 4.4-2 update to fix this.

SirTux · September 24, 2019, 9:07pm

The upgrade works also if the folders are removed before.

damrose · September 25, 2019, 8:13am

Thanks for reporting the issue and identifying a cause for the failing update. We have just added a check to our pre-update checks and will block the update for now if the directory /var/lib/docker/aufs exists.

A fix for the issue will be done at Bug 50261

eljot · October 1, 2019, 2:52pm

Will there be an email notice when the bug is cleared and the update will work from 4.4.1 to 4.4.2?

damrose · October 1, 2019, 3:25pm

To stay updated you can register an account in our bugzilla and add yourself to the bug’s CC list. I will also try to remember to post an update to this thread once the bug is CLOSED

msi · October 3, 2019, 8:12am

@damrose

Please consider clearly communicating the end of the maintenance cycle for UCS 4.4-1 once this bug gets closed (via updating your release Blog, this forum or whatever fits best). As by Univention’s definition, 4.4-1 would not receive any security-related bugfixes 6 weeks after 4.4-2 has been released, this would be 6th November and updating to 4.4-2 becomes mandatory in order to receive security updates.

This blocker effectively shrinks the overlapping time between when 4.4-1 and 4.4-2 would receive critical security bugfixes. From a customer perspective I’d appreciate if 4.4-1 would stay maintained until this bug is fixed and that the 6 weeks of overlap would be reset at that moment. - But that’s just my very own opinion

Nonetheless: Clearly communicating how this blocker affects the end of maintenance for 4.4-1 helps system administrators to plan accordingly.

Edit: Typo.

gulden · November 6, 2019, 11:47am

Hello,

FYI: Because of update issue regarding UCS 4.4-2 and the necessary time to fix it, we have decided to extend the end of security maintenance for UCS 4.4-1 until 03. Dec. 2019.

Best regards,
Nico

msi · November 7, 2019, 9:33am

Hi Nico, thanks for the update and the - definitely appreciated - extension of the EoL date of UCS 4.4-1.

gulden · November 7, 2019, 10:12am

Hello @msi,

you are welcome. Thank you for the hint on this and it certainly put some weight into our internal discussion on this.

Best regards,
Nico