Slides from the talk "CI doesn’t start with Jenkins" from DevOps Stage 2018 (12-13 October 2018, Kyiv, Ukraine) CI is not only a tool. You cannot simply install and configure Jenkins or whatever system and say that you have CICD pipeline. Here I'm trying to cover different aspects and dependencies of the CICD process based on Preply Inc experience. External Links: [2] CatOps Telegram channel: https://t.me/catops [2] HashiCorp User Group Kyiv: https://www.meetup.com/Kyiv-HashiCorp-User-Group/ [12-24 ]https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work [49-50]: https://nvie.com/posts/a-successful-git-branching-model/ [51-56]: https://www.toptal.com/software/trunk-based-development-git-flow [64] Django Anonymizer: https://github.com/knowledge-point/dj_anonymizer

1. CI doesn’t start with Jenkins
DevOps Stage - 2018: October 12-13

2. Bio
•CatOps Engineer @ Preply Inc
•Twitter: @grem11n
•Telegram: @grem1in
• CatOps
• Kyiv HashiCorp User Group

3. Agenda
Pain
CICD evolution @ Preply
External dependencies for CI
Tools
Cats

4. Let’s start with Jenkins, tho

6. In the very beginning

7. In the very beginning

8. In the very beginning
Separate teams

9. In the very beginning
Separate teams Monolith

10. Welcome, Jenkins!

11. Dependencies
• A tool

12. Skullcandy flow (code)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. Create a pull request
to the master

13. Skullcandy flow (code)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. Create a pull request
to the master
2. Label it once ready

14. Skullcandy flow (code)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. Create a pull request
to the master
2. Label it once ready
3. Gather all “ready
PRs” into ephemeral
branch

15. Skullcandy flow (code)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. Create a pull request
to the master
2. Label it once ready
3. Gather all “ready
PRs” into ephemeral
branch
4. Test it!

16. Skullcandy flow (code)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. Create a pull request
to the master
2. Label it once ready
3. Gather all “ready
PRs” into ephemeral
branch
4. Test it!
5. Merge QA branch to
the master

17. Skullcandy flow (artifact)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. QA artifact is created
from the ephemeral
QA branch

18. Skullcandy flow (artifact)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
1. QA artifact is created
from the ephemeral
QA branch
2. Production artifact is
created after QA
branch is merged to
master

19. Skullcandy flow (pros)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• We can live with the
monolith

20. Skullcandy flow (pros)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• We can live with the
monolith
• Release candidates
every day

21. Skullcandy flow (pros)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• We can live with the
monolith
• Release candidates
every day
• Works pretty well with
GitHub

22. Skullcandy flow (pros)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• We can live with the
monolith
• Release candidates
every day
• Works pretty well with
GitHub
• Clear priorities

23. Skullcandy flow (cons)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• PRs could have
conflicts with each
other

24. Skullcandy flow (cons)
https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work
• PRs could have
conflicts with each
other
• You need to wait QA
before resolve the
conflicts

25. Dependencies
• A tool
• Development flow

26. Feel the pain [1]
• Unknown status of PR

27. Feel the pain [1]
• Unknown status of PR
• Lack of notifications

28. Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD

29. Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work

30. Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work
• Unknown time for QA

31. Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work
• Unknown time for QA
• Unstable tests

32. Dependencies
• A tool
• Development flow
• Alignment

33. Dependencies
• A tool
• Development flow
• Alignment
• Sane tests

34. Skullcandy 2.0
• GitHub API

35. Skullcandy 2.0
• GitHub API
• Merge constraints

36. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership

37. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership
• Codestyle

38. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership
• Codestyle
• Sane tests

39. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership
• Codestyle
• Sane tests
• UI tests

40. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership
• Codestyle
• Sane tests
• UI tests
• QA session limits

41. Skullcandy 2.0
• GitHub API
• Merge constraints
• Code ownership
• Codestyle
• Sane tests
• UI tests
• QA session limits
• More notifications

42. Dependencies
• A tool
• Development flow
• Alignment
• Sane tests
• UI tests

43. Dependencies
• A tool
• Development flow
• Alignment
• Sane tests
• UI tests
• Discipline

44. So, are we done?

45. Feel the pain [2]
• Conflicts between PRs

46. Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA

47. Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA
• You cannot merge
directly

48. Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA
• You cannot merge
directly
• Hotfixes are harmful

49. Gitflow?
https://nvie.com/posts/a-successful-git-branching-model/
• Everybody likes Gitflow
• Easy to automate
• Clear enough

50. Gitflow
https://nvie.com/posts/a-successful-git-branching-model/
• Everybody likes Gitflow
• Easy to automate
• Clear enough
• Broken development
branch is a company-
wide problem

51. Trunk-based development!
https://www.toptal.com/software/trunk-based-development-git-flow
• Only one mainline
• Clear releases (tags)
• Everyone is independent
• Hotfixes are the same entities

52. Trunk-based development
https://www.toptal.com/software/trunk-based-development-git-flow
1. Feature branch is created
and tested separately

53. Trunk-based development
https://www.toptal.com/software/trunk-based-development-git-flow
1. Feature branch is created
and tested separately
2. Feature is merged directly
into the master when it’s
ready

54. Trunk-based development
https://www.toptal.com/software/trunk-based-development-git-flow
1. Feature branch is created
and tested separately
2. Feature is merged directly
into the master when it’s
ready
3. New tag is created from the
master branch, which goes
to production

55. Trunk-based development
https://www.toptal.com/software/trunk-based-development-git-flow
1. Each feature is a separate
artifact for QA

56. Trunk-based development
https://www.toptal.com/software/trunk-based-development-git-flow
1. Each feature is a separate
artifact for QA
2. Production artifact is created
from tag

57. Wait, but …
• What if trunk is broken?

58. Wait, but …
• What if trunk is broken?
• Test better!

59. Wait, but …
• What if trunk is broken?
• Test better!
• Where can I test?!

60. Wait, but …
• What if trunk is broken?
• Test better!
• Where can I test?!
• …

61. Welcome, Kubernetes!

62. Welcome, Kubernetes!
You cannot just put Kubernetes to fix everything!11
Me:

63. Dynamic env for each PR
• Kubernetes
• Helm
• Jenkins

64. DB for each environment!
Django Anonymizer:
• Preply & OpenSource
• Utilizes Django ORM
• Safe data for development

65. Dependencies
• A tool
• Development flow
• Alignment
• Sane tests
• Discipline
• UI testes
• Infrastructure

66. Deal with it!

67. Let’s add some services!
• Monorepo or not?
• Repeatable CI
• Env for the whole thing

68. And we also have Lambdas
• CI for Lambdas
• LocalStack
• Docker

69. And we also have Lambdas
• CI for Lambdas
• LocalStack
• Docker
• CD for Lambdas
• Serverless framework
• Terraform

70. Dependencies
• A tool
• Development flow
• Alignment
• Sane tests
• Discipline
• UI testes
• Infrastructure
• Architectural planning

71. Dependencies
• A tool (tech, ops)
• Development flow (tech, mgmt)
• Alignment (cult, mgmt)
• Sane tests (tech, dev)
• Discipline (cult, all)
• UI testes (tech, qa)
• Infrastructure (tech, ops)
• Architectural planning (tech, all)

72. A Toolset

73. A Toolset
Jenkins:
• Multibranch project
• Declarative pipeline
• Shared libraries

74. A Toolset
Jenkins:
• Multibranch project
• Declarative pipeline
• Shared libraries
Shared libraries are
pretty much Groovy :(

75. A Toolset
Dynamic Env:
• Kubernetes on Spot
• Helm

76. A Toolset
Dynamic Env:
• Kubernetes on Spot
• Helm
• Chart for entire environment
is complex
• Helm 2 vs Helm 3

77. Where are we going to?

78. Where are we going to?
• Stable CICD flow for existing monolith
• Survive in the Microservices world
• Figure out deploy to k8s
(to Helm or not to Helm)
• Canary (Blue/Green) deployments
• Perhaps try JenkinsX
• Do something with CICD for Lambdas

79. Thank you!
yrochnyak@gmail.com
TG: @grem1in && @catops
Questions?