Makefiles in 2020 — Why they still matter

Makefiles in 2020
Why they still matter

Release Engineer @trivago_tech

✋
projects with multiple languages

If you have worked in a company with many different
projects for long enough …

Congratulations 🎂
You now work on a new*
project!

Congratulations 🎂
You now work on a new*
project!
*
new for you

abandoned project as a dependency

you adopt a shiny, new puppy 🐶

it’s 1 AM 🕐
the website is broken 🧨
you need to release a fix 🚒

$ ls
README.md
cf-templates/
cloudtrail_crawler.service
event_rules/
inventory_crawler.service
inventory_crawler.timer
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
How do you build that?

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
python3 ./setup.py bdist_wheel

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
How do you test that?

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
python3 ./setup.py test

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
python3 -m pytest

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
python3 -m pytest
tox

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
How do you install that?

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
pip3 install --upgrade ./
cp -r ./{event,inventory}_rules
/etc/resourcemanagement
install ./inventory_crawler.timer
/etc/systemd/system/inventory_crawler.timer
install ./inventory_crawler.service
/etc/systemd/system/inventory_crawler.service
install ./cloudtrail_crawler.service
/etc/systemd/system/cloudtrail_crawler.service
useradd -r resourcemanagement
systemctl daemon-reload
How do you install that?

$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini
How do you deploy that?

$ ls cf-templates/
EC2Machine.yaml
MonitoringRoleCreation.yaml
$ ls
README.md
cf-templates/
event_rules/
inventory_rules/
resourcemanagement/
setup.py
tests/
tox.ini

$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
php72-compatibility-fix.patch
postcss.config.js
sonar-project.properties
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

composer install
composer test
npm ci
npm test
$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

DIST_DIR=$(pwd)/dist
mkdir -p "${DIST_DIR}"
composer install --no-interaction --no-dev --optimize-autoloader --profile
npm ci
npm run build
# custom commands
S_CONSOLE="php -d memory_limit=2048M ./app/${APP_NAME}/console"
${S_CONSOLE} cache:clear --no-warmup --env=prod --no-debug
${S_CONSOLE} cache:warmup --env=prod --no-debug
${S_CONSOLE} assets:install --env=prod --no-debug ./web/"${RELEASE_VERSION}$
{RELEASE_SUBVERSION}"
${S_CONSOLE} company:asset-dump --env=prod --no-debug
${S_CONSOLE} company:container-dump --env=prod --no-debug
# remove stuff we don't want in our packages'
rm web/*_dev.php web/*_integration.php web/*_jenkins.php
rm -rf node_modules test-js test test-trupi
# package PHP
tar --exclude=".svn" --exclude=".git" --exclude=".tar.gz" -czf
"${DIST_DIR}/${PACKAGE_ID}.tar.gz" *
# package PHP cache
pushd /var/www/prod
find ${PACKAGE_ID} -type d -exec chmod 777 {} +
"${DIST_DIR}/${PACKAGE_ID}.cache.tar.gz" ${PACKAGE_ID}
popd
# package assets
cd ./web
“${DIST_DIR}/${PACKAGE_ID}.assets.tar.gz" v*
$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

DIST_DIR=$(pwd)/dist
mkdir -p "${DIST_DIR}"
composer install --no-interaction --no-dev --optimize-autoloader --profile
npm ci
npm run build
# custom commands
S_CONSOLE="php -d memory_limit=2048M ./app/${APP_NAME}/console"
${S_CONSOLE} cache:clear --no-warmup --env=prod --no-debug
${S_CONSOLE} cache:warmup --env=prod --no-debug
${S_CONSOLE} assets:install --env=prod --no-debug ./web/"${RELEASE_VERSION}$
{RELEASE_SUBVERSION}"
${S_CONSOLE} company:asset-dump --env=prod --no-debug
${S_CONSOLE} company:container-dump --env=prod --no-debug
# remove stuff we don't want in our packages'
rm web/*_dev.php web/*_integration.php web/*_jenkins.php
rm -rf node_modules test-js test test-trupi
# package PHP
"${DIST_DIR}/${PACKAGE_ID}.tar.gz" *
# package PHP cache
pushd /var/www/prod
find ${PACKAGE_ID} -type d -exec chmod 777 {} +
"${DIST_DIR}/${PACKAGE_ID}.cache.tar.gz" ${PACKAGE_ID}
popd
# package assets
cd ./web
“${DIST_DIR}/${PACKAGE_ID}.assets.tar.gz" v*
Obviously 🙄
$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

Good luck with that 🍀
$ ls
CHANGELOG.md
Dockerfile
Patches/
README.md
app/
bin/
composer.json
composer.lock
package-lock.json
package.json
postcss.config.js
src/
src-js/
test/
test-trupi/
web/
webpack.config.js

README
Everybody can read it, many won’t

CI/CD pipeline
continuous delivery for free*

CI/CD pipeline
continuous delivery for free*
*
not really

CI/CD pipeline
everybody can use it*

CI/CD pipeline
everybody can use it*
*
if they know the URLs

CI/CD pipeline
debugging/fixing pipelines 🔥

language specific tools
great for pros

OK for beginners

and then you add a 2nd language

Use a language agnostic build tool

CMake, redo, Gradle, Meson, SCons, Ninja, ant, Bazel, tup, …

CMake, redo, Gradle, Meson, SCons, Ninja, ant, Bazel, tup, …
✋

Let’s start with a simple build.sh

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
tar czf dist/$1.tar.gz web/

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
no -x

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
no -e

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
no tests

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
needs an argument

#!/bin/bash
composer install
--no-interaction
--no-dev
npm ci
npm run build
needs an argument
weird issues

#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
echo "PACKAGE_ID missing"
return 1
else
export PACKAGE_ID="$1"
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
tar czf dist/"$PACKAGE_ID".tar.gz
web/
vendor/

#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/
a bit verbose

not simple to just run the tests
#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/

re-installs all npm packages
#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/

more and more cases
#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/

more and more scripts
#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/

good luck 🍀
#!/usr/bin/env bash
set -ex
if [ -n "$1" ]; then
return 1
else
fi
composer install
--no-interaction
--no-dev
composer test
npm ci
npm run build
npm test
web/
vendor/

But why Makefiles?
execute steps in the shell

But why Makefiles?
we already have most of that

But why Makefiles?
-ex for free*

But why Makefiles?
-ex for free*
*
for real this time

But why Makefiles?
installed virtually everywhere

But why Makefiles?
dependencies for steps

But why Makefiles?
file based dependencies

But why Makefiles?
handbook of common tasks

But why Makefiles?
very useful at 1 AM 🕐

But why Makefiles?
or with a new puppy 🐶

test: dependencies
composer test
npm test
# alias for the actual Artifact
build: dist/my_project.tar.gz
dist/my_project.tar.gz:
npm run build
mkdir -p dist/
tar czf dist/my_project.tar.gz web/ vendor/
dependencies:
composer install --no-interaction --no-dev
npm install
.PHONY: test build dependencies

test: dependencies
composer test
npm test
npm run build
mkdir -p dist/
dependencies:
npm install
test: dependencies
composer test
npm test

test: dependencies
composer test
npm test
npm run build
mkdir -p dist/
dependencies:
npm install
npm run build
mkdir -p dist/

test: dependencies
composer test
npm test
npm run build
mkdir -p dist/
dependencies:
npm install
dependencies:
npm install

test: test-php test-js
test-php: dependencies-php
composer test
test-js: dependencies-js
npm test
npm run build
mkdir -p dist/
dependencies: dependencies-php dependencies-js
dependencies-php:
dependencies-js:
npm install
.PHONY: test test-php test-js build dependencies
dependencies-php dependencies-js
dependencies-php:
dependencies-js:
npm install

composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
composer test
npm test

composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
npm run build
mkdir -p dist/

composer test
npm test
build: dist/${PACKAGE_ID}.tar.gz
dist/${PACKAGE_ID}.tar.gz:
npm run build
mkdir -p dist/
tar czf dist/"${PACKAGE_ID}".tar.gz web/ vendor/
dependencies-php:
dependencies-js:
npm install
npm run build
mkdir -p dist/

composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
npm run build
mkdir -p dist/
$ make build PACKAGE_ID=my_package-1.2.9

composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
$ make build
npm run build
mkdir -p dist/

composer test
npm test
dist/${PACKAGE_ID}.tar.gz: env-PACKAGE_ID
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
# guard against being called without parameters
# or environment variables
env-%:
test -n "${$*}" 
(echo "Argument $* not set"; exit 1)
dependencies-php dependencies-js env-%
[…]
env-%:
test -n "${$*}" 

composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
env-%:
@test -n "${$*}" 
env-%:
@test -n "${$*}" 

composer test
npm test
npm run build
mkdir -p dist/
tar czf $@ web/ vendor/
dependencies-php:
dependencies-js:
npm install
env-%:
@test -n "${$*}" 
npm run build
mkdir -p dist/

VERSION := $(shell git rev-parse --short HEAD)
PACKAGE_ID := “my_package-${VERSION}"
composer test
npm test
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
PACKAGE_ID := "my_package-${VERSION}"

DOCKER_REGISTRY := "docker.company.com/"
DOCKER_NAME := "my-project:${VERSION}"
composer test
npm test
docker: dist/${PACKAGE_ID}.tar.gz
docker build
--build-arg PACKAGE=$<
-t "${DOCKER_REGISTRY}${DOCKER_NAME}"
./
push: docker
docker push "${DOCKER_REGISTRY}${DOCKER_NAME}"
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
dependencies-php dependencies-js docker push
DOCKER_NAME := “my-project:${VERSION}"
docker build
./
push: docker

composer test
npm test
docker build
./
push: docker
deploy: push
kubectl apply -f k8s/deployment.yaml
npm run build
mkdir -p dist/
dependencies-php:
dependencies-js:
npm install
.PHONY: test test-php test-js build dependencies dependencies-php
dependencies-js docker push
deploy: push
kubectl apply -f k8s/deployment.yaml

composer test
npm test
docker build
./
push: docker
deploy: k8s/deployment.yaml push
kubectl apply -f $<
npm run build
mkdir -p dist/
k8s/deployment.yaml: k8s/deployment.yaml.template
sed 's#my-project:VERSION#${DOCKER_NAME}#'
> $@
dependencies-php:
dependencies-js:
npm install
.PHONY: test test-php test-js build dependencies dependencies-php
dependencies-js docker push k8s/deployment.yaml
kubectl apply -f $<
> $@

composer test
npm test
docker build
./
push: docker
kubectl apply -f $<
npm run build
mkdir -p dist/
> $@
clean:
rm -rf ./vendor/ # php packages
npm clean
rm k8s/deployment.yaml
distclean:
rm dist/*.tar.gz
dependencies-php:
dependencies-js:
npm install
.PHONY: test test-php test-js build dependencies dependencies-php dependencies-js docker push k8s/deployment.yaml clean distclean
clean:
npm clean
distclean:
rm dist/*.tar.gz

composer test
npm test
docker build
./
push: docker
kubectl apply -f $<
npm run build
mkdir -p dist/
> $@
clean:
npm clean
distclean:
rm dist/*.tar.gz
dependencies-php:
dependencies-js:
npm install
In your CI/CD system:
$ make test deploy

composer test
npm test
docker build
./
push: docker
kubectl apply -f $<
npm run build
mkdir -p dist/
> $@
clean:
npm clean
distclean:
rm dist/*.tar.gz
dependencies-php:
dependencies-js:
npm install
On your development system:
$ make test

composer test
npm test
docker build
./
push: docker
kubectl apply -f $<
npm run build
mkdir -p dist/
> $@
clean:
npm clean
distclean:
rm dist/*.tar.gz
dependencies-php:
dependencies-js:
npm install
At 1 AM 🕐 to release a fix 🚒:
$ make deploy

end;
@m3t0r
https://www.gnu.org/software/make/manual/make.html

common Makeﬁle targets
libraries applications docker-based projects
all ✅ ✅ ✅
build ✅ ✅ ✅
debug ✅ ✅ ✅
test ✅ ✅ ✅
release ✅ ✅
push ✅
deploy ✅ ✅
clean ✅ ✅ ✅
distclean ✅ ✅
install ✅
up ✅
see make manual — 16.6 Standard Targets for Users

- FOO = "value" 
The value is recursively computed every time FOO is being
accessed.
- FOO := "value" 
The value is directly computed and assigned once to FOO.
- FOO ?= "value" 
The value is lazily set if FOO has no value at the moment it
is being accessed.
- FOO += "value" 
The value is appended to FOO with a space in front.
see make manual — 6.2 The Two Flavors of Variables
The different kinds of assignments

As an argument (better):
$ make target FOO=bar
As an environment variable (requires ?= in Makefile):
$ FOO=bar make target
Setting variables on make invocation
see make manual — 9.5 Overriding Variables

Dos
- Fan out from general targets to more specific ones: deploy " deploy-nomad, not deploy-
nomad " deploy TOOL=nomad, use pattern rules to accomplish this with similar targets.
- Split up big targets into smaller chunks; that gives the user the possibility to not
have to run everything but only the parts the they are interested in: test " test-unit,
test-integration, test-codestyle, then the user can just call make test-unit.
- Name targets after files where-ever possible. It will reduce the amount of unnecessary
computation for invocations.
- Use variables to abstract common command calls. ${MAKE} and ${CC} are standards. Why not
add ${GREP} or ${DOCKER}, so users can point your Makefile to the binary they want to
use for those programs?

Don’ts
- Use :: to define phony rules. That’s just a side-effect of their main purpose (to split
one target over multiple Makefiles).
- Add to many targets that aren’t used on a regular basis. They will become stale without
you noticing.

Makefiles in 2020 — Why they still matter

More Related Content

Similar to Makefiles in 2020 — Why they still matter

Recently uploaded

Makefiles in 2020 — Why they still matter