Phạm Quang Minh - A real DevOps culture in practice

1. A real DevOps culture
in practice
PHAM QUANG MINH
@MONMEN
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 1

2. About me
• Pham Quang Minh
◦ @monmen
• 7 years being a Dev
• 7 years being an Ops
• Used to be:
◦ Data Engineer
◦ Cloud Engineer
◦ Solution Architect
• Well, call me a DevOps?
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 2

3. Agenda
• The Dev – Ops battle
• DevOps to the “rescue”
• A true DevOps environment
• FAQ
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 3

4. The Dev – Ops battle
WITHOUT GUNS
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 4

5. Dev vs Ops
• Just pull my code
• Heil master branch
• My local still working
• My cronjob run twice
• I need some-strange-db
on prod tomorrow
• No, dockerize it
• Dev, Stg please
• But prod is not!
• It’s called: Cluster
• Wait, what?
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 5

6. Ops vs Dev
• Smth eat all db conn
& I don’t know why
• Resource?
• Is your code alive?
• I don’t know, either
• 16 CPU and 32GB
ram like mine?
• Is your server dead?
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 6

7. Dev vs Ops battle
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 7

8. The S.W.A.T. team
• Developer with Ops skills
• Have access to prod system
• Master in debugging
• Deep knowledge about
application
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 8

9. DevOps to the “rescue”
IT MAKES YOU THINK SO!
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 9

10. DevOps
10
August 27, 2022 OPENINFRA DAYS VIETNAM 2022

11. My DevOps journey
• 50 VMs, 15 micro-services
◦ Full IaC with Terraform + Ansible
◦ K8S, auto scaling, auto DNS, SSL,…
◦ CI/CD, auto testing
◦ 10 mins provisioned from scratch
• Took me 6 months
• No release
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 11

12. A million ways to die in DevOps
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 12

13. Challenges
13
• App runtime
• Ops mindset
• System integration
• Docker
• Language
• App logic
• App architecture
August 27, 2022 OPENINFRA DAYS VIETNAM 2022

14. A lot of confusion
• = Dev + Container
• = Sys + CI/CD
• A team or position?
• A bunch of tools?
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 14

15. Lost in CNCF
15
August 27, 2022 OPENINFRA DAYS VIETNAM 2022

16. Should we change?
16
August 27, 2022 OPENINFRA DAYS VIETNAM 2022

17. A true DevOps
environment
IT’S HARDER THAN YOU THINK!
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 17

18. The hardest part
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 18

19. Common language
• https://12factor.net
• Migration rule
• DevOps review
◦ Architecture
◦ Resource planning
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 19

20. The Procedure
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 20

21. Planning
• Architecture
• Dependencies
◦ DB, cache, message queue,…
• Sizing
◦ Resource, workload plan
• Git flow
◦ Branch, environment
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 21

22. Coding
• The 12 factors
◦ Config: env, file,…
◦ Logging: stdout, stderr, log level,…
◦ Healthcheck
◦ Concurrency: cluster, ha,…
◦ Disposability: OS signal handling,…
◦ Tracing: error tracing,…
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 22

23. Deploying
• Deploy request: what, when, where,…
• Release note: changes
• Affected services
• Migration steps: zero downtime or not?
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 23

24. Operating
• Centralize logs
• Resource monitoring: CPU, RAM,…
• CI/CD dashboard: Deployment info
• Error tracing
• Notification: deployment, downtime, warning
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 24

25. Key notes
• Involve ASAP
• Transparent info
• Sharing
• Avoid tool race
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 25

26. Q&A
ASK ME ANYTHING
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 26

27. Thanks for watching
August 27, 2022 OPENINFRA DAYS VIETNAM 2022 27