Benjamin Bischoff presents on simplifying test runs with Make. He discusses how Make is a proven build automation tool that has been used for decades. While new tools can be tempting, Make offers benefits like flexibility, consistency across environments, and simplicity. Bischoff demonstrates how to structure a basic Makefile to build targets in sequence and handle dependencies between files. He also provides a real-world example of using Make to run Selenium tests across different browsers and devices.

1. Benjamin Bischo
ff
| @bischo
ff
dev | softwaretester.blog
Simplifying your test runs with „Make“
TestMμ Conference 2022

2. Benjamin Bischo
ff
• Test Automation Engineer
• 7 years of SDET experience
• ~ 22 years in IT overall
• 6 years at trivago N.V.
About me
@bischo
ff
dev

3. Credits
“Make
fi
les in 2019 - Why They Still Matter”
Simon Brüggen
trivago Tech Blog
2019
“Das nützlich-unbedenklich Spektrum”
(“The useful harmless spectrum“)
Felix von Leitner,
2019
@bischo
ff
dev

4. The Prologue
@bischo
ff
dev

5. „Architect of destruction“
Barney Stinson:
“I’m gonna give you four words to live by:
new is always better! [...]”
Ted Mosby:
“And this theory applies to everything?”
Barney Stinson:
“Everything.”
@bischo
ff
dev

6. Is new always better?
@bischo
ff
dev

7. Developers
love new things.
@bischo
ff
dev

8. Learning new
things is crucial.
@bischo
ff
dev

9. The "New is always better" dilemma
• Familiarity
Getting comfortable with new solutions takes time.
• Maturity
New software can have
fl
aws and unwanted dependencies.
• Management
Management may view a proof of concept as close to
fi
nal.
• Tractor Factor
A single developer can be the only one in the know.
@bischo
ff
dev

10. If you learn a new tool,
you want to use it.
@bischo
ff
dev

11. The Psychology of Science, Abraham Maslow, 1966
„[...] I suppose it is tempting, if the only tool you
have is a hammer, to treat everything as if it were a
nail.“
@bischo
ff
dev

12. What is Make?
@bischo
ff
dev

13. Make (software), Wikipedia
„[...] Make is applicable to any process that involves
executing arbitrary commands to transform a
source file to a target result.“
@bischo
ff
dev

14. Make
A few facts
• Invented by Stuart Feldman in 1976 at Bell Labs
• Initially a build automation tool for executable programs
• Built natively into Unix, Linux, MacOS
• Windows: Chocolatey, Cygwin, Make for Windows or WSL
@bischo
ff
dev

15. Make
Cons
• Can be hard to master
• Dependent on tabs and spaces
• Bugs can be tricky to
fi
gure out
@bischo
ff
dev

16. Make
Pros
• Basically available everywhere
• Proven solution
• Consistent across local and CI builds
• Easy to learn
• Very
fl
exible
• Never out of sync with documentation
@bischo
ff
dev

17. Parts of a Makefile
@bischo
ff
dev

18. Makefile
Naming and invocation
• make is the name of the tool
• It looks for make
fi
le or Make
fi
le to run
• It returns a status of 0 if successful
make [ -f makefile ] [ options ] … [ targets ] ...
@bischo
ff
dev

19. Makefile
Rules
• A make rule typically creates (and checks) a target
fi
le
• It is mostly dependent on timestamps
• It can also be the name of an action without a target
fi
le
@bischo
ff
dev

20. Makefile
Recipes
• A recipe is an action that make invokes
• Can be one or multiple commands
• Careful: each line has to be started with a tab!
@bischo
ff
dev

21. Makefile
Rule & Recipe
dependency.txt:
@echo "Hello world" > dependency.txt
@echo "File created"
@bischo
ff
dev

22. The Prerequisites
@bischo
ff
dev

23. Makefile
Prerequisites
final.txt: dependency.txt
echo "This is the final file!" > final.txt
cat dependency.txt >> final.txt
dependency.txt:
echo "Hello world" > dependency.txt
@bischo
ff
dev

24. This sounds .Phony
@bischo
ff
dev

25. Makefile
.Phony
final.txt: dependency.txt
echo "This is the final file!" > final.txt
cat dependency.txt >> final.txt
dependency.txt:
echo "Hello world" > dependency.txt
@bischo
ff
dev

26. Makefile
.Phony
final.txt: dependency.txt
echo "This is the final file!" > final.txt
cat dependency.txt >> final.txt
dependency.txt: clean
echo "Hello world" > dependency.txt
clean:
rm -f dependency.txt
rm -f final.txt
@bischo
ff
dev

27. Makefile
.Phony
final.txt: dependency.txt
echo "This is the final file!" > final.txt
cat dependency.txt >> final.txt
dependency.txt: clean
echo "Hello world" > dependency.txt
.PHONY: clean
clean:
rm -f dependency.txt
rm -f final.txt
@bischo
ff
dev

28. When does it build what?
@bischo
ff
dev

29. Makefile
Build order
clean:
@echo "Clean up old files"
build: clean
@echo "Build application"
deploy: build
@echo "Deploy application to stage server"
test: deploy
@echo "Test stage deployment"
.PHONY: clean build deploy test
@bischo
ff
dev

30. Makefile
Build order
clean:
@echo "Clean up old files"
build: clean
@echo "Build application"
deploy: build
@echo "Deploy application to stage server"
test: deploy
@echo "Test stage deployment"
.PHONY: clean build deploy test
@bischo
ff
dev

31. Makefile
Build order
clean:
@echo "Clean up old files"
build: clean
@echo "Build application"
deploy: build
@echo "Deploy application to stage server"
test: deploy
@echo "Test stage deployment"
.PHONY: clean build deploy test
@bischo
ff
dev

32. Makefile
Build order
test: deploy
@echo "Test stage deployment"
deploy: build
@echo "Deploy application to stage server"
build: clean
@echo "Build application"
clean:
@echo "Clean up old files"
.PHONY: clean build deploy test
@bischo
ff
dev

33. Makefile
Build order
test: deploy
@echo "Test stage deployment"
deploy: build
@echo "Deploy application to stage server"
build: clean
@echo "Build application"
clean:
@echo "Clean up old files"
.PHONY: clean build deploy test
@bischo
ff
dev

34. Makefile
Build order
test: deploy
@echo "Test stage deployment"
deploy: build
@echo "Deploy application to stage server"
build: clean
@echo "Build application"
clean:
@echo "Clean up old files"
.PHONY: clean build deploy test
@bischo
ff
dev

35. Makefile
Build order
test: deploy
@echo "Test stage deployment"
deploy: build
@echo "Deploy application to stage server"
build: clean
@echo "Build application"
clean:
@echo "Clean up old files"
.PHONY: clean build deploy test
@bischo
ff
dev

36. Makefile
Build order
test: deploy
@echo "Test stage deployment"
deploy: build
@echo "Deploy application to stage server"
build: clean
@echo "Build application"
clean:
@echo "Clean up old files"
.PHONY: clean build deploy test
@bischo
ff
dev

37. What about parameters?
@bischo
ff
dev

38. Makefile
Parameters
BROWSER ?= firefox
test:
@echo "Testing with browser ${BROWSER}"
.PHONY: deploy test
@bischo
ff
dev

39. Just help yourself!
@bischo
ff
dev

40. Makefile
Tip: help
# Show this help.
help:
@awk '/^#/{c=substr($$0,3);next}c&&/^[[:alpha:]]
[[:alnum:]_-]+:/{print substr($$1,1,index($
$1,":")),c}1{c=0}' $(MAKEFILE_LIST) | column -s: -t
# Test project.
test:
@echo "This would start a test"
.PHONY: help test
@bischo
ff
dev

41. Makefile
Example code
https://github.com/bischo
ff
dev/make_examples
@bischo
ff
dev

42. This is not too complex
@bischo
ff
dev

43. Stuart Feldman, „The Art of Unix Programming“, Eric S. Raymond, 2003
„[...] Makefiles were text files, not magically encoded
binaries, because that was the Unix ethos:
printable, debuggable, understandable stuff.“
@bischo
ff
dev

44. Limitation as an
advantage
@bischo
ff
dev

45. Real world example
Selenium tests with Maven invocation
mvn verify -e -B -DargLine=-Dfile.encoding=UTF-8 -Dproject=„${PROJECT_NAME}”
-Dwebdriver.driver="${browserName}” -Dwebdriver.driver.version="${browserVersion}”
-Dtrupi.browser.resolution="${BROWSER_RESOLUTION}" -Dwebdriver.base.url="${BASE_URL}"
-Dtrupi.feature="${FEATURES}" -Dcustom.activated.ctests=„${CTESTS}" -Dwebdriver.hub.url="${HUB_URL}"
-Dparallel.fork.count=„${PARALLEL_RUNS}" -Dcucable.test.runs.count="${TEST_RUNS}"
-Dinclude.scenario.tags=„${INCLUDE_TAGS}" -Dtrupi.restart.browser="${RESTART_BROWSER}"
-Dtrupi.webdriver.wait=„${WEBDRIVER_WAIT}" -Dtrupi.pageload.wait="${PAGELOAD_WAIT}"
-Dtrupi.pageload.retries="${PAGELOAD_RETRIES}" -Dtrupi.kafka.enabled="${KAFKA_LOGGING_ENABLED}"
-Dtrupi.kafka.label="${KAFKA_LABEL}" -Dtrupi.logging="${LOG_MODE}"
-Dtrupi.plugin.rerun.detector.rules.file=„${RERUN_RULES_LOCATION}“ -
Dtrupi.plugin.browserstack.capabilityset.enabled=„${PLUGIN_BROWSERSTACK_ENABLED}“ -
Dtrupi.plugin.browserstack.capabilityset.browserstack.local=„${BROWSERSTACK_LOCAL}"
-Dtrupi.plugin.browserstack.capabilityset.device=„${deviceName}"
-Dtrupi.plugin.browserstack.capabilityset.realMobile="${isRealDevice}"
-Dtrupi.plugin.browserstack.capabilityset.os=„${osName}"
-Dtrupi.plugin.browserstack.capabilityset.os_version="${osVersion}"
-Dtrupi.plugin.browserstack.failurepost.enabled="${PLUGIN_BROWSERSTACK_ENABLED}"
-Dtrupi.plugin.selenoid.capabilities.enabled="${PLUGIN_SELENOID_ENABLED}"
@bischo
ff
dev

46. Real world example
Selenium tests with Make invocation
make test-local
BASE_URL=https://www.trivago.com
BROWSERS="chrome"
@bischo
ff
dev

47. Real world example
Selenium tests with Make invocation
make test-grid
BASE_URL=https://www.trivago.com
BROWSERS="chrome"
@bischo
ff
dev

48. Real world example
Selenium tests with Make invocation
make test-grid
BASE_URL=https://www.trivago.com
BROWSERS="chrome, firefox"
DEVICES="Samsung Galaxy"
FEATURES="src/test/resources/features/Search.feature"
@bischo
ff
dev

49. Real world example
Invocation in a GitHub action
- name: Run tests
id: runUiTests
working-directory: ./tests
run: >
make test-grid
BASE_URL="${{ github.event.inputs.baseUrl }}"
BROWSERS="${{ steps.setBrowser.outputs.browserString }}"
FEATURES="${{ github.event.inputs.features }}"
@bischo
ff
dev

50. Just a sidenote
@bischo
ff
dev

51. Just
A few facts
• Heavily inspired by make
• It is only a command runner (no .Phony needed)
• Runs on Linux, MacOS and Windows
• Needs to be installed on the host system/container
@bischo
ff
dev

52. Justfile
Rule & Recipe
myRecipe: someDependency
@echo "Hello from Just"
someDependency:
@echo "I am a dependency"
@bischo
ff
dev

53. The Epilogue
@bischo
ff
dev

54. Every project is different.
@bischo
ff
dev

55. Use the right tool for the job!
@bischo
ff
dev

56. Decouple progress
from technology!
@bischo
ff
dev

57. Testing is a
team effort.
@bischo
ff
dev

58. Listen to your
customers.
@bischo
ff
dev

59. Hide the
complexity!
@bischo
ff
dev

60. „Architect of destruction“
Barney Stinson:
“I’m gonna give you four words to live by:
new is always better! […]”
Ted Mosby:
“And this theory applies to everything?”
Barney Stinson:
“Everything.”
@bischo
ff
dev
Original version

61. „Architect of destruction“
Revised version
Barney Stinson:
“I’m gonna give you
fi
ve words to live by:
new is not always better! [...]”
Ted Mosby:
“I agree!”
@bischo
ff
dev

62. Thank you, TestMμ!
@bischo
ff
dev | softwaretester.blog