file deduplication in docker?

dbadude

How can i make sure 10 docker containers use deduplication on storage level?
I dont want to use 10 times the storage when i install 1 application in all 10 docker instances.

tentor

https://docs.docker.com/engine/storage/drivers/#sharing-promotes-smaller-images

sillycat

Any thoughts on my 3.4GB docker images? Thanks.

dbadude

@tentor said:
https://docs.docker.com/engine/storage/drivers/#sharing-promotes-smaller-images

so if you pull the images with the binaries already installed then it will be deduplicated. so the installation of binaries after pulling, is not as effictient as it creates different copies as the metadata timestamps is not persee the same.

rm_

If you store it on a filesystem supporting reflinks and copy on write, then you can just use a program like jdupes in crontab to deduplicate it from time to time.

Such filesystems are Btrfs and XFS. Also ZFS, but that's so much more involved to get into.

ahmadpiran

@dbadude You nailed it. if you run the install command inside the running container, it writes to the unique writable layer of each instance, so you end up using 10x the storage.
The cleanest fix is to just build a custom image (Dockerfile) with the app pre-installed. Docker automatically shares the read-only layers, so 10 containers will still only take up the space of 1 image.
If you can't rebuild the image for some reason, you could just bind mount the application folder from the host into all the containers so they share the same physical files. Way easier than messing with ZFS or filesystem dedupe!