Lessons learned from doing it repeatedly by Mirah Gary & Josh Hill, Pivotal Pipelines are the infrastructure we use to test and deliver our code. We work with them every day, but sometimes they behave in unexpected ways. We will go over common failure modes and issues we have encountered in our Concourse pipelines, and understand what we can do to prevent and solve these pain points. Being able to build pipelines that work for you is critical for effective dev/ops. By sharing our own mistakes, we hope that you will be able to recognize them when they happen and know what to do to fix the problems. This talk was delivered at the Concourse London User Group (CLUG) on 6 December 2018 at Pivotal London: https://www.meetup.com/Concourse-London-User-Group/events/256171643/

1. How NOT to build a pipeline
Lessons learned from doing it wrong, repeatedly
Mirah Gary & Josh Hill
Pivotal

2. Overview
Build Antipatterns
- by Mirah
Flow Antipatterns
- by Josh

3. Builds

4. Builds pass ✅
Builds fail ❌
Builds error ⚠

5. Build Errors
Symptom: concourse error
Visual: orange build
Diagnosis: misconfiguration
Treatment: correct the pipeline definition

6. Symptom: git resource failing due to API rate limit
Treatment: create a GitHub concourse user & authenticate
resources:
- name: my-github-repo
type: git
source:
uri: git@github.com:me/my-github-repo.git
private_key: ((git-private-key))
branch: master
GitHub API rate limit

7. Error Here
run

8. Continuing on Failure
Symptom: pipeline fails due to a failed build in an earlier job
Treatment: tasks should fail when errors occur
tasks are code, test them like code

9. Bad Bash
Treatment: Shellcheck
set -eu
set -o pipefail

11. Undebuggable Tasks
Symptom: task fails with no useful output
Treatment: add logging to stdout/stderr
Gotcha: set +x can leak creds

12. Flakes
Symptom: job fails for no reason, passes when run again
Visual:
Treatment: hunt down and fix flakes, make yourself feel the pain

13. Copypasta
Symptom: many nearly identical tasks
Visual: tasks
├── run-systests-mysql-5.5
│ ├── task.sh
│ └── task.yml
├── run-systests-mysql-5.6
│ ├── task.sh
│ └── task.yml
└── run-systests-mysql-5.7
├── task.sh
└── task.yml
Treatment: refactor tasks

14. Input Mapping
Lets you rename inputs, so tasks have general, reusable inputs
plan:
- aggregate:
- get: cf-deployment-concourse-tasks
- get: my-private-repo
- task: bbl-up-my-BOSH-director
file: cf-deployment-concourse-tasks/bbl-up/task.yml
input_mapping:
bbl-state: my-private-repo/my-state-dir

15. Copypasta Take 2
Symptom: job with incorrect passed constraints
Visual:
Treatment: visually inspect after set-pipeline

16. Tasks with Side Effects
Symptom: running a task again after failure always fails again
unless manually cleaned up
Example: tasks which push to GitHub
Treatment: Idempotent tasks
Tasks are not done until they can be run repeatedly,
even if they fail
State should live in resources, not tasks

18. No clean up
Symptom: Job fails because a limit is reached
Visual: failed build
Treatment: automate clean up

19. Flow

20. Pipelines can flow ✅

22. Pipelines can block ❌

23. Pipelines can nonsense .

24. Plumbing

27. No pipe
Symptom: job has to be run before another job can pass
Visual: jobs not linked
Treatment: get resource & add passed constraint

28. No pipe
Symptom: job has to be run before another job can pass
Visual: jobs not linked
Treatment: get resource & add passed constraint
jobs:
- name: system-director
plan:
- get: dummy-bbr-script-release-bucket
passed: [deploy-bosh-directors]

31. Missing trigger
Symptom: job does not trigger automatically on previous job
Visual: dotted line between jobs
Treatment: add trigger

32. Missing trigger
Symptom: job does not trigger automatically on previous job
Visual: dotted line between jobs
Treatment: add trigger
jobs:
- name: system-director
plan:
- get: dummy-bbr-script-release-bucket
passed: [deploy-bosh-directors]
trigger: true

33. Fans

38. Incomplete fan in
Symptom: version that fails tests is delivered
Visual: lines inconsistent with other jobs
Treatment: add passed constraint

40. Overlapping fan out
Symptom: too many builds in the pipeline
Visual: too many lines
Treatment: minimise passed constraints

41. Resources

45. build image
build image
again!

46. build image
once

47. Wrong resource flows
Symptom: rebuilding artifacts in the pipeline
Visual: wrong resource flows through
Treatment: change resource that flows through

48. deploy test delete

49. deploy test delete

50. 1. deploy
2. test
4. delete
3. deploy
5. test

51. Missing resource
Symptom: triggering serial group twice causes failed build
Visual: failed build in serial group
Treatment: add resource to represent state

52. Missing resource
Symptom: triggering serial group twice causes failed build
Visual: failed build in serial group
Treatment: add resource to represent state

53. Summary

54. Pipeline anti-patterns
Builds
● Build errors
● API rate limiting
● Continuing on failure
● Bad bash
● Undebuggable tasks
● Flakes
● Copypasta
● Tasks with side effects
● No clean up
Flow
● No pipe
● Missing trigger
● Incomplete fan in
● Overlapping fan out
● Wrong resource flows
● Missing resource

55. That’s all!
Pipelines: github.com/cloudfoundry-incubator/backup-and-restore-ci
Concourse: backup-and-restore.ci.cf-app.com

56. Thank you .
Mirah Gary & Josh Hill
Pivotal