The document discusses challenges with caching dependencies and sources when building Docker images across different environments. It finds that builds are faster when caching locally but slower when caching dependencies across CI/CD pipelines due to differences in file permissions and generated files. Specifically: 1) File permissions differ between local builds and CI/CD due to user and group settings 2) Generated files like documentation and cache files cause issues because they are not ignored 3) Reinstalling all dependencies from scratch on each build is slow. It provides solutions like fixing group permissions, setting dockerignore, pre-building wheels, and multi-stage builds to better leverage caching across environments.

1. The hardest thing
in computer science

2. Hard things

3. Docker Caching
Dependency versions
Install dependencies
[ 20 minutes or so ]
Only here copy all sources

4. Intended behaviour
● No change:
docker is not rebuilt - LIGHTNING FAST!!!!
● Sources change/dependencies not:
only sources are added - QUITE FAST !!!
● Dependencies change:
dependencies installed, sources - LITTLE SLOWER !!

5. Actual behaviour
same machine - local checkout
● Local docker registry
● Repeated build: 1:06m
● Only sources: 1:30m
● Dependencies: 11m
● Whole build: ~ 30m

6. CI case
● Always fresh machine
○ no code
○ no registry
● Git clone/checkout
● Build
● Wipeout

7. Docker registry to the rescue!
Build cache:
● Docker build
● Docker push airflow/airflow:latest
Use cache:
● Docker pull airflow/airflow:latest
● docker build --cache-from ariflow/airflow:latest

8. Actual behaviour
Docker Hub automated build
● DockerHub docker registry as cache
● Repeated build: 11m
● Only sources: 11m <- Still OK
● Dependencies: ~1h
● Whole build: ~ 2h

9. Using the cache in Travis CI
● Docker Hub builds are slow
● Travis or Cloud Build use earlier image
with --cache-from
● But only sources change most of the
time

11. Actual BAD behaviour
Travis CI automated build
● Build on Travis with cache from DockerHub
● Repeated build: 11m
● Only sources: 1 h <-
● Dependencies: 1h
● Whole build: ~ 2h

13. Problem no 1
Git & permissions
● git clone file creation:
○ local user
○ default user’s group
● file/dir permissions (rwxs)
○ preserves user, group and other rx permissions files & dirs
○ does not store w and by default uses umask when cloning by default
○ core.sharedRepository git-config
■ one of: group(true), all, umask(false), 0xxx
● Umask WTF:
○ file: 644 (DockeHub) vs. 664 (Travis CI)
○ dir: 755 (DockerHub) vs. 775 (Travis CI)

14. Solution to problem 1
Fix group permissions

15. Problem no 2
Generated files
● not only .gitignore
● generated files
○ autoapi - documentation
○ build artifacts
○ npm cache
○ .pyc files
○ files created accidentally (wget in source folder anyone?)
● COPY .
● Context calculated based on ALL files
● .dockerignore != .gitignore
● slightly different syntax

16. Solution to problem 2
Set .dockerignore ** by default

17. Problem no. 3
● Download & compile ALL dependencies takes time!

18. Partial solution to problem 3
Find the weakest link

19. Solution to problem 3
a) build image with wheels

20. Solution to problem 3
b) Copy directory via multi-stage
Docker builds

21. Solution 3
c) install using wheels

23. Thank You!
You can add some info where to follow you,
or add information about
polidea.com/blog