Massive growth in Docker Overlay directory

Apps & App Center
1

Good morning all,

I’ve noticed a massive growth within the Docker Overlay directory (/var/lib/docker/overlay/), its currently at about 22GB in size. Regarding to “docker ps -s” there are only 3 active containers running on my system.

I have tried already “docker system prune -a -f” in order to free up some disk space, but this seems to have no effect at all.

So I’m wondering what is the right approach with UCS to keep this directory maintained and free up disk space again? Some hints or instructions would be much appreciated.

Best regards
Thomas

2

Hey,

as directories from /var/lib/docker/overlay itself are mounted underneath /var/lib/docker/overlay, did you add the -x parameter to du? E.g. du -shx /var/lib/docker/overlay instead of just du -sh /var/lib/docker/overlay? Might give a wrong impression on how much space is really used by counting files more than once if -x wasn’t used.

m.

3

Hi Moritz,

thx for your quick reply!

interesting finding …

du -shx /var/lib/docker/overlay gives me:
4,0G    /var/lib/docker/overlay

As UCS is running here in a virtual machine (Hyper-V), I have noticed first that there is a rapid growing of the according vhdx-file.Trying to find out what is eating up the disk space I ran “df -h”, whitch gave me:

Dateisystem             Größe Benutzt Verf. Verw% Eingehängt auf
udev                      10M       0   10M    0% /dev
tmpfs                    1,6G     16M  1,6G    1% /run
/dev/mapper/vg_ucs-root  233G     22G  200G   10% /
tmpfs                    3,9G     64K  3,9G    1% /dev/shm
tmpfs                    5,0M       0  5,0M    0% /run/lock
tmpfs                    3,9G       0  3,9G    0% /sys/fs/cgroup
/dev/sda1                472M     62M  387M   14% /boot
overlay                  233G     22G  200G   10% /var/lib/docker/overlay/b8897c4f3f88716d851afbc5588cddea4bfc4848a276c7e884aef5e7d20c4f88/merged
shm                       64M       0   64M    0% /var/lib/docker/containers/2666d313af9fdc4536336015b489e115aa2e3f08ea2d2a56170b6fd7cddc5a5c/shm
overlay                  233G     22G  200G   10% /var/lib/docker/overlay/c8f9b46ecd22347ba03106e731fba8cca822dbdaefc4b95f86094e769a479d3b/merged
shm                       64M       0   64M    0% /var/lib/docker/containers/8c42ed6b764c4a3ae5d026f72c9f9b04a20a41e6e590d862423dca8f34c20e40/shm
overlay                  233G     22G  200G   10% /var/lib/docker/overlay/baaa409fac8eae23b31d741a22d845c0bac1cde2535228e234e6238c78608abc/merged
shm                       64M       0   64M    0% /var/lib/docker/containers/531a0b2ff059136e1fcbd242f7aea03869d1a76ad69e150625191fd4569d8219/shm

Obviously this was a bit misleading, though I’m not sure how this can be interpreted :slight_smile:
Thanks for your support anyway.

Best regards
Thomas

4

Hey,

what’s mounted on the /var/lib/docker/overlay mount point is the root file system (device /dev/mapper/vg_ucs-root). Therefore the 22GB used space are across your whole root device — which isn’t all that unusual.

If you’re curious where all the space is used, I highly suggest you try the ncdu utility (you’ll have to enable the unmaintained repository before you can install it). It’s an ncurses-based, interactive tool for digging into space usage on a file system. Run it like this:

ncdu -x /

Kind regards,
mosu

5

Hi mosu,

thanks a lot for clarification!
Next I will install ncdu and give it a try :slight_smile:

Best regards
Thomas

6

AFAIR vhdx files do not perfectly shrink when you delete files from the underlying filesystem, so if you perform a lot of filesystem operations in the virtual machine the vhdx file will continue to grow.

7

Hi crbble,

thx for your feedback, I found this out too in the meanwhile, but running “compress” vhdx-file from the Hyper-V manager does not show much effect. I seems in order to get a good result (compression) it is required to “zero out” the unused blocks on the file system first. I don’t have a clue how to achieve this with UCS.

Any hints are much appreciated :slight_smile:

Best regards
Thomas

8

/edit: Sorry, I just saw that this thread is already quite old.

Hey, in case you are comfortable with the command line you can simply create files that contain only zeros, for example:

dd if=/dev/zero of=./testfile bs=1024 count=1024

This reads from /dev/zero, outputs to a file named testfile in the local directory. The blocksize is 1024 (if you need huge files try to increase this value to some multiple of 1024) and count is 1024, so in this case you write a file with 1048576 bytes. Increase the count value to the filesize you want. Leave some disk space unused, otherwise your system may crash/stop, especially when running this command as root. Do not forget to remove the temporary file afterwards.

9

thx crbble for your feedback. Will give it a try later.